=head1 NAME

Parse::StringTokenizer - Extract fields from strings
No symbols are exported by default.



=head1 SYNOPSIS


  use Parse::StringTokenizer;
  my $tokenizer = String::Tokenizer->new();
  my $str = 'a b c';
  die unless 'a' eq $tokenizer->shift($str);
  die unless 'b c' eq $str;



=head1 EXPORTS

Nothing exported by default

=over



=back

=head1 DEPENDENCIES

This module requires these other modules and libraries:

    Error::Programatic
    Perl::Module

=head1 DESCRIPTION


Similiar in spirit to C<strtok>.

Default delimiter is whitespace (regex C<\s>).

Default quotes which protect the delimiter are: C<'">




=head1 PUBLIC INTERFACE

=head2 new

=over

=item Constructor




options:


  -delim => $delimiter      # Token separator
  -quotes => $quotes        # Quote characters which protect the
                            # delimiter, and will be ignored
  -contained => $chars      # Character pairs which protect the delimiter
  -keywords => \@keywords   # Words which will not be split
  -preserve => 1            # Preserve quotes which surround each field

By default, the following is considered three tokens:


  one 'and a two' "and a three"




=back


=head2 split

=over

=item Unpack the string and return non-empty fields






=back


=head2 unpack

=over

=item Split a string into fields (opposite of pack)






=back


=head2 pack

=over

=item Join fields together as a string (opposite of unpack)




-preserve

  TODO: when packing the field will be double quoted if it contains the
  primary delimiter.  If the field is quoted, then do not double-quote



=back


=head2 shift

=over

=item Return the first field and trim the string






=back


=head2 pop

=over

=item Return the last field and trim the string






=back


=head2 push

=over

=item Push a field on to the string



Returns the new field count.



=back


=head1 PACKAGE INTERNALS

=head2 _compile

=over

=item Compile the regular expression




Quotes and contained segments can have escaped characters, like:


  'don\'t break me'

right now, it comes back as

  dont\'t break me

TODO, unescape it so it becomes

  dont't break me

the difficulty is that you DO NOT want to unescape the backslashes here

  "my quotes \'are\' different"

which means reworking the RE such that the leading and tailing quotes are 
NOT removed, i.e., they remain part of the segment.  and a subsequent 
step added to `unpack` which knows what to do by the presence of leading and
trailing characters.



=back



=head1 AUTHORS

    Ryan Gies <ryangies@cpan.org>

=head1 COPYRIGHT

    Copyright (C) 2014-2016 by Ryan Gies. All rights reserved.
    Copyright (C) 2006-2013 by Livesite Networks, LLC. All rights reserved.
    Copyright (C) 2000-2005 by Ryan Gies. All rights reserved.
    Redistribution and use in source and binary forms, with or without 
    modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright notice, 
    this list of conditions and the following disclaimer.
    * The origin of this software must not be misrepresented; you must not 
    claim that you wrote the original software. If you use this software in a 
    product, an acknowledgment in the product documentation would be 
    appreciated but is not required.
    * Altered source versions must be plainly marked as such, and must not be 
    misrepresented as being the original software.
    * The name of the author may not be used to endorse or promote products 
    derived from this software without specific prior written permission.

    THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED 
    WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 
    MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO 
    EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 
    EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT 
    OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 
    INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
    CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 
    IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY 
    OF SUCH DAMAGE.
    To the best of our knowledge, no patented algorithms have been used. However, we
    do not have the resources to carry out a patent search, and therefore cannot 
    give any guarantee of the above statement.

=cut

