Class CodeRay::Scanners::Scanner
In: lib/coderay/scanner.rb
Parent: StringScanner

Scanner

The base class for all Scanners.

It is a subclass of Ruby‘s great StringScanner, which makes it easy to access the scanning methods inside.

It is also Enumerable, so you can use it like an Array of Tokens:

  require 'coderay'

  c_scanner = CodeRay::Scanners[:c].new "if (*p == '{') nest++;"

  for text, kind in c_scanner
    puts text if kind == :operator
  end

  # prints: (*==)++;

OK, this is a very simple example :) You can also use map, +any?+, find and even sort_by, if you want.

Methods

Included Modules

Enumerable

Constants

ScanError = Class.new(Exception)   Raised if a Scanner fails while scanning
DEFAULT_OPTIONS = { :stream => false }   The default options for all scanner classes.

Define @default_options for subclasses.

External Aliases

string -> code
  More mnemonic accessor name for the input string.

Public Class methods

[Source]

    # File lib/coderay/scanner.rb, line 72
72:         def file_extension extension = nil
73:           if extension
74:             @file_extension = extension.to_s
75:           else
76:             @file_extension ||= plugin_id.to_s
77:           end
78:         end

Create a new Scanner.

  • code is the input String and is handled by the superclass StringScanner.
  • options is a Hash with Symbols as keys. It is merged with the default options of the class (you can overwrite default options here.)
  • block is the callback for streamed highlighting.

If you set :stream to true in the options, the Scanner uses a TokenStream with the block as callback to handle the tokens.

Else, a Tokens object is used.

[Source]

     # File lib/coderay/scanner.rb, line 106
106:       def initialize code='', options = {}, &block
107:         @options = self.class::DEFAULT_OPTIONS.merge options
108:         raise "I am only the basic Scanner class. I can't scan "\
109:           "anything. :( Use my subclasses." if self.class == Scanner
110: 
111:         super Scanner.normify(code)
112: 
113:         @tokens = options[:tokens]
114:         if @options[:stream]
115:           warn "warning in CodeRay::Scanner.new: :stream is set, "\
116:             "but no block was given" unless block_given?
117:           raise NotStreamableError, self unless kind_of? Streamable
118:           @tokens ||= TokenStream.new(&block)
119:         else
120:           warn "warning in CodeRay::Scanner.new: Block given, "\
121:             "but :stream is #{@options[:stream]}" if block_given?
122:           @tokens ||= Tokens.new
123:         end
124: 
125:         setup
126:       end

[Source]

    # File lib/coderay/scanner.rb, line 66
66:         def normify code
67:           code = code.to_s
68:           code.force_encoding 'binary' if code.respond_to? :force_encoding
69:           code.to_unix
70:         end

Returns if the Scanner can be used in streaming mode.

[Source]

    # File lib/coderay/scanner.rb, line 62
62:         def streamable?
63:           is_a? Streamable
64:         end

Public Instance methods

code=(code)

Alias for string=

[Source]

     # File lib/coderay/scanner.rb, line 182
182:       def column pos = self.pos
183:         return 0 if pos <= 0
184:         pos - (string.rindex(?\n, pos) || 0)
185:       end

Traverses the tokens.

[Source]

     # File lib/coderay/scanner.rb, line 167
167:       def each &block
168:         raise ArgumentError,
169:           'Cannot traverse TokenStream.' if @options[:stream]
170:         tokens.each(&block)
171:       end

The current line position of the scanner.

Beware, this is implemented inefficiently. It should be used for debugging only.

[Source]

     # File lib/coderay/scanner.rb, line 178
178:       def line
179:         string[0..pos].count("\n") + 1
180:       end

[Source]

     # File lib/coderay/scanner.rb, line 128
128:       def reset
129:         super
130:         reset_instance
131:       end

Whether the scanner is in streaming mode.

[Source]

     # File lib/coderay/scanner.rb, line 162
162:       def streaming?
163:         !!@options[:stream]
164:       end

[Source]

     # File lib/coderay/scanner.rb, line 133
133:       def string= code
134:         code = Scanner.normify(code)
135:         super code
136:         reset_instance
137:       end

Scans the code and returns all tokens in a Tokens object.

[Source]

     # File lib/coderay/scanner.rb, line 144
144:       def tokenize new_string=nil, options = {}
145:         options = @options.merge(options)
146:         self.string = new_string if new_string
147:         @cached_tokens =
148:           if @options[:stream]  # :stream must have been set already
149:             reset unless new_string
150:             scan_tokens @tokens, options
151:             @tokens
152:           else
153:             scan_tokens @tokens, options
154:           end
155:       end

[Source]

     # File lib/coderay/scanner.rb, line 157
157:       def tokens
158:         @cached_tokens ||= tokenize
159:       end

Protected Instance methods

Scanner error with additional status information

[Source]

     # File lib/coderay/scanner.rb, line 213
213:       def raise_inspect msg, tokens, state = 'No state given!', ambit = 30
214:         raise ScanError, "\n\n***ERROR in %s: %s (after %d tokens)\n\ntokens:\n%s\n\ncurrent line: %d  column: %d  pos: %d\nmatched: %p  state: %p\nbol? = %p,  eos? = %p\n\nsurrounding code:\n%p  ~~  %p\n\n\n***ERROR***\n\n" % [
215:           File.basename(caller[0]),
216:           msg,
217:           tokens.size,
218:           tokens.last(10).map { |t| t.inspect }.join("\n"),
219:           line, column, pos,
220:           matched, state, bol?, eos?,
221:           string[pos - ambit, ambit],
222:           string[pos, ambit],
223:         ]
224:       end

[Source]

     # File lib/coderay/scanner.rb, line 207
207:       def reset_instance
208:         @tokens.clear unless @options[:keep_tokens]
209:         @cached_tokens = nil
210:       end

This is the central method, and commonly the only one a subclass implements.

Subclasses must implement this method; it must return tokens and must only use Tokens#<< for storing scanned tokens!

[Source]

     # File lib/coderay/scanner.rb, line 202
202:       def scan_tokens tokens, options
203:         raise NotImplementedError,
204:           "#{self.class}#scan_tokens not implemented."
205:       end

Can be implemented by subclasses to do some initialization that has to be done once per instance.

Use reset for initialization that has to be done once per scan.

[Source]

     # File lib/coderay/scanner.rb, line 194
194:       def setup
195:       end

[Validate]