Class | CodeRay::Scanners::Scanner |
In: |
lib/coderay/scanner.rb
|
Parent: | StringScanner |
The base class for all Scanners.
It is a subclass of Ruby‘s great StringScanner, which makes it easy to access the scanning methods inside.
It is also Enumerable, so you can use it like an Array of Tokens:
require 'coderay' c_scanner = CodeRay::Scanners[:c].new "if (*p == '{') nest++;" for text, kind in c_scanner puts text if kind == :operator end # prints: (*==)++;
OK, this is a very simple example :) You can also use map, +any?+, find and even sort_by, if you want.
ScanError | = | Class.new(Exception) | Raised if a Scanner fails while scanning | |
DEFAULT_OPTIONS | = | { :stream => false } |
The default options for all scanner classes.
Define @default_options for subclasses. |
string | -> | code |
More mnemonic accessor name for the input string. |
# File lib/coderay/scanner.rb, line 72 72: def file_extension extension = nil 73: if extension 74: @file_extension = extension.to_s 75: else 76: @file_extension ||= plugin_id.to_s 77: end 78: end
If you set :stream to true in the options, the Scanner uses a TokenStream with the block as callback to handle the tokens.
Else, a Tokens object is used.
# File lib/coderay/scanner.rb, line 106 106: def initialize code='', options = {}, &block 107: @options = self.class::DEFAULT_OPTIONS.merge options 108: raise "I am only the basic Scanner class. I can't scan "\ 109: "anything. :( Use my subclasses." if self.class == Scanner 110: 111: super Scanner.normify(code) 112: 113: @tokens = options[:tokens] 114: if @options[:stream] 115: warn "warning in CodeRay::Scanner.new: :stream is set, "\ 116: "but no block was given" unless block_given? 117: raise NotStreamableError, self unless kind_of? Streamable 118: @tokens ||= TokenStream.new(&block) 119: else 120: warn "warning in CodeRay::Scanner.new: Block given, "\ 121: "but :stream is #{@options[:stream]}" if block_given? 122: @tokens ||= Tokens.new 123: end 124: 125: setup 126: end
# File lib/coderay/scanner.rb, line 66 66: def normify code 67: code = code.to_s 68: code.force_encoding 'binary' if code.respond_to? :force_encoding 69: code.to_unix 70: end
# File lib/coderay/scanner.rb, line 182 182: def column pos = self.pos 183: return 0 if pos <= 0 184: pos - (string.rindex(?\n, pos) || 0) 185: end
Whether the scanner is in streaming mode.
# File lib/coderay/scanner.rb, line 162 162: def streaming? 163: !!@options[:stream] 164: end
# File lib/coderay/scanner.rb, line 133 133: def string= code 134: code = Scanner.normify(code) 135: super code 136: reset_instance 137: end
Scans the code and returns all tokens in a Tokens object.
# File lib/coderay/scanner.rb, line 144 144: def tokenize new_string=nil, options = {} 145: options = @options.merge(options) 146: self.string = new_string if new_string 147: @cached_tokens = 148: if @options[:stream] # :stream must have been set already 149: reset unless new_string 150: scan_tokens @tokens, options 151: @tokens 152: else 153: scan_tokens @tokens, options 154: end 155: end
Scanner error with additional status information
# File lib/coderay/scanner.rb, line 213 213: def raise_inspect msg, tokens, state = 'No state given!', ambit = 30 214: raise ScanError, "\n\n***ERROR in %s: %s (after %d tokens)\n\ntokens:\n%s\n\ncurrent line: %d column: %d pos: %d\nmatched: %p state: %p\nbol? = %p, eos? = %p\n\nsurrounding code:\n%p ~~ %p\n\n\n***ERROR***\n\n" % [ 215: File.basename(caller[0]), 216: msg, 217: tokens.size, 218: tokens.last(10).map { |t| t.inspect }.join("\n"), 219: line, column, pos, 220: matched, state, bol?, eos?, 221: string[pos - ambit, ambit], 222: string[pos, ambit], 223: ] 224: end
# File lib/coderay/scanner.rb, line 207 207: def reset_instance 208: @tokens.clear unless @options[:keep_tokens] 209: @cached_tokens = nil 210: end
This is the central method, and commonly the only one a subclass implements.
Subclasses must implement this method; it must return tokens and must only use Tokens#<< for storing scanned tokens!
# File lib/coderay/scanner.rb, line 202 202: def scan_tokens tokens, options 203: raise NotImplementedError, 204: "#{self.class}#scan_tokens not implemented." 205: end