Writing a very simple lexical analyser in C++

This is a quick and dirty solution iterating on each pattern, and for each pattern trying to match the entire string, then iterating over matches and storing each match with its position in a map. The map implicitly sorts the matches by key (position) for you, so then you just need to iterate the map to get the matches in positional order, regardless of their pattern name.

#include <iterator>
#include <iostream>
#include <string>
#include <regex>
#include <list>
#include <map>

using namespace std;

int main(){

    string str = " hello how are 2 * 3 you? 123 4567867*98";

    // define list of patterns
    map<string,string> patterns {
        { "[0-9]+" ,   "NUMBERS" },
        { "[a-z]+" ,   "IDENTIFIERS" },
        { "\\*|\\+",  "OPERATORS" }
    };

    // storage for results
    map< size_t, pair<string,string> > matches;

    for ( auto pat = patterns.begin(); pat != patterns.end(); ++pat )
    {
        regex r(pat->first);
        auto words_begin = sregex_iterator( str.begin(), str.end(), r );
        auto words_end   = sregex_iterator();

        for ( auto it = words_begin; it != words_end; ++it )
            matches[ it->position() ] = make_pair( it->str(), pat->second );
    }

    for ( auto match = matches.begin(); match != matches.end(); ++match )
        cout<< match->second.first << " " << match->second.second << endl;
}

Output:

hello IDENTIFIERS
how IDENTIFIERS
are IDENTIFIERS
2 NUMBERS
* OPERATORS
3 NUMBERS
you IDENTIFIERS
123 NUMBERS
4567867 NUMBERS
* OPERATORS
98 NUMBERS

Related Posts:

  1. OR condition in Regex
  2. What is a Context Free Grammar?
  3. What does \’.- mean in a Regular Expression
  4. How do I fix a “Expected Primary-expression before ‘)’ token” error?
  5. How can I exclude one word with grep?
  6. How can I validate an email address using a regular expression?
  7. Regular expression to match a line that doesn’t contain a word
  8. Difference between \b and \B in regex
  9. Regex: ignore case sensitivity
  10. What does the regex \S mean in JavaScript?
  11. How can I validate an email address using a regular expression?
  12. Parse (split) a string in C++ using string delimiter (standard C++)
  13. Regex: ignore case sensitivity
  14. Regular Expression for alphanumeric and underscores
  15. What is the difference between the regular expressions [^\d\s] and [\D\S]
  16. Regular Expression for alphanumeric and underscores
  17. grep –ignore-case –only
  18. Regex AND operator
  19. python re.split() to split by spaces, commas, and periods, but not in cases like 1,000 or 1.50
  20. python re.split() to split by spaces, commas, and periods, but not in cases like 1,000 or 1.50
  21. Regular expression negative lookahead
  22. Regular Expression to match string starting with a specific word
  23. Regular Expressions: Is there an AND operator?
  24. Regular expression to match standard 10 digit phone number
  25. Regular Expression.how to add optional character at end of regex
  26. Regular Expression with wildcards to match any character
  27. What is a non-capturing group in regular expressions?
  28. How to validate an email address in PHP
  29. What is a non-capturing group in regular expressions?
  30. Regular expression to match standard 10 digit phone number
  31. How to use Regular Expressions (Regex) in Microsoft Excel both in-cell and loops
  32. Python string.replace regular expression
  33. How to use Regular Expressions (Regex) in Microsoft Excel both in-cell and loops
  34. Python string.replace regular expression
  35. Symbol for any number of any characters in regex?
  36. Grep and Python
  37. Regex – Does not contain certain Characters
  38. How to replace ” \ ” with ” \\ ” in java
  39. A regular expression to exclude a word/string
  40. Regex how to match an optional character
  41. How to validate an email address in JavaScript
  42. Using Bash regex match (=~) where regex includes quotes (” characters)
  43. Searching for UUIDs in text with regex
  44. Regular expression to match a word or its prefix
  45. How can I remove punctuation from input text in Java?
  46. Regular expression for exact match of a string
  47. Using the star sign in grep
  48. Regex optional group
  49. Regex for string contains?
  50. Regular Expression – Validate Gmail addresses
  51. Java; String replace (using regular expressions)?
  52. How can I remove punctuation from input text in Java?
  53. How can I output only captured groups with sed?
  54. How to negate specific word in regex?
  55. What is the grep equivalent in Python?
  56. javascript HTML from document.body.innerHTML
  57. Which regular expression operator means ‘Don’t’ match this character?
  58. Python re.split() vs split()
  59. Negative matching using grep (match lines that do not contain foo)
  60. How to exclude a specific string constant?
  61. How to replace a whole line with sed?
  62. Validating IPv4 addresses with regexp
  63. How to validate phone number using PHP?
  64. Validate phone number with JavaScript
  65. Can a URL contain a semicolon and still be valid?
  66. How is \\n and \\\n interpreted by the expanded regular expression?
  67. Regex to match only uppercase “words” with some exceptions
  68. Regular Expression to reformat a US phone number in Javascript
  69. Regex to match only letters
  70. Regex optional capturing group?
  71. Finding the complement of a DFA?
  72. RegEx to find two or more consecutive chars
  73. Regular expressions in C: examples?
  74. How to ignore whitespace in a regular expression subject string?
  75. Regex: matching up to the first occurrence of a character
  76. Regex credit card number tests
  77. Regular expression – starting and ending with a character string
  78. How do I match any character across multiple lines in a regular expression?
  79. Regular expression to limit number of characters to 10
  80. Regex that matches integers in between whitespace or start/end of string only
  81. Regular Expressions- Match Anything
  82. What is the difference between square brackets and parentheses in a regex?
  83. Regex for “AND NOT” operation
  84. Regular expression that matches valid IPv6 addresses
  85. Escaping Parentheses in Regex
  86. How to replace a string in an existing file in Perl
  87. What is the best regular expression to check if a string is a valid URL?
  88. Regex empty string or email
  89. Regex difference: (\w+)? and (\w*)
  90. Wrap element with in any number of elements with PHP
  91. extract shortcodes from string
  92. preg_replace and comment_form_defaults
  93. How to set up redirects for category paginated pages after adding /category/ to permalink
  94. Adding span tags to post titles using regex
  95. Redirect non www to www using redirection plugin regex
  96. Restrict comments with regular expression
  97. Call Shortcode Attribute Value within another function
  98. How to fix old embeded iframes with WP-CLI search-replace
  99. Regex For Extracting First 4 letters of search term
  100. Use Regex Redirection via htsaccess for RSS Feeds WordPress

Leave a Comment