I like using syntax highlighting when it breaks. For example, if all code below a particular line is a single color, then I probably forgot an end-quote or something. But this has become less uniquely useful due to the broader integration of parser-driven linters (e.g. tree-sitter), which—besides being able to drive highlighting—can explicitly deliver inline hints or parsing errors.
All that said, I'm one who appreciates information density! How about coloring branching code paths/call stacks?
My keyboard has a concept of "layers," which allows each key to map differently depending on the layer. I've seen this used to make a numpad or to have a QWERTY and DVORAK layer. What if highlighting was the same? Instead of competing for priority over the color channels, developers could explicitly swap layers?
> All that said, I'm one who appreciates information density! How about coloring branching code paths/call stacks?
>
> My keyboard has a concept of "layers," which allows each key to map differently depending on the layer. I've seen this used to make a numpad or to have a QWERTY and DVORAK layer. What if highlighting was the same? Instead of competing for priority over the color channels, developers could explicitly swap layers?
I was thinking about coloring logic /scope blocks as a way to help visualize scope and flow, even if it required static analysis and a simple script it could be useful when I need to debug
For the last five years I've been working on this problem!
To solve it we need to be able to describe the structured content of a document without rendering it, and that means we need an embedding language for code documents.
I hope this doesn't sound overly technical: I'm just borrowing ideas from web browsers. I think of my project as being the creation of a DOM for code documents. The DOM serves a similar function. A semantic HTML documents has meaning independent of its rendered presentation and so it can be rendered many ways.
CSTML is my novel embedding language for code. You could think of it like a safe way to hold or serialize an arbitrary parse tree. Like HTML a CSTML document has "inner text" which this case is the source text if the program the parser saw. E.g. a tiny document might be `<Boolean> 'true' </>`. The parser injects node tags into the source text, creating what is essentially the perfect data stream to feed a syntax highlighter. To do the highlighting you print the string content if the document and use the control tags to decide on color. This is actually already how we syntax highlight the output from our own CLI as it happens. We use our streaming parser technology to parse our log output into a CSTML tag stream (in real time) and then we just swap out open and close node tags for ANSI escape codes, print the strings, and send that stream to stdout.
Having this in the editor leads to people removing that information from other channels. For example class properties are now often coloured, so people do not name or refer to them to denote that they are not some random other variable. I don't like this, because I think all information should be in the code itself and it can be really annoying once you work and haven't your favorite editor available in the correct setup.
with the same argument i used to reject syntax highlighting in general. all the information should already be in the code. if i can't read that, then the syntax of the language is bad. syntax highlighting would just make me lazy. i have since decided that i like being lazy so i do like to use it, but you make a good point, if the use of colors makes you not write something in the code that you would otherwise, then that's not good. i can see that with syntax highlighting too, where the colors help me read code even if it is badly formatted, so i am less inclined to format for readability.
True, but I haven't experienced that for general syntax highlighting. I do read the code through cat or in other contexts often enough that I haven't lost the ability to read code at normal speed without syntax highlighting. Do you have an example?
i wasn't talking about the ability to read code, even though that was my initial argument for rejecting colors. what i mean is that without colors i might take more effort to write/format code so that it is more readable, instead of relying on syntax highlighting:
with colors this is perfectly readable, because if, for and return appear in red, and other keywords in blue. so they stand out, making the structure more visible than without colors.
> Color carries a huge amount of information. Color draws our attention. Color distinguishes things. And we just use it to distinguish syntax.
I didn't actually find the uncoloured circle to be that much harder to spot.
Before looking at the concrete examples, I thought:
> Understanding the syntax similarly isn't hard scanning over the code. But if the words were coloured in a way that didn't correspond to syntax, I expect I would find it distracting, in the same way as those experiments with colour words written in a different colour from what the word indicates. Trying to make use of that second information channel isn't necessarily a good idea.
> I like the aesthetic of syntax-coloured code. But also, it's reassuring. I'm not using it to help understand the syntax, but to confirm that there's software in place that understands the syntax the same way I do.
After: the rainbow parentheses honestly don't help that much, not for a language like LISP anyway. The context highlighting seems to work much better. But then, as I go on through the examples... these are all really doing the same kind of thing that syntax highlighting does! They're about the structure of the code. And I'd have to shift my expectation, but once aligned, it's again something I'd perceive the same way — as something to confirm my understanding rather than eliciting that understanding.
Perhaps being able to switch between colouring modes would change that, but I don't know how quickly I could get used to that technique. And then, the kind of code where most of these things would help is smelly anyway. I already try to avoid this kind of nesting. Maybe import and argument highlighting (which could be used at the same time, along with highlighting for class attributes and closures in Python?), though...
Many (most?) of these are achievable with neovim and tree-sitter (a plugin that gives you access to the AST), and surely many other editors. I have plugins installed right now that do several of the things that are mocked up here. Many more are done with virtual text and not color, but I don’t see why you couldn’t use highlighting instead.
I agree with the broader point of the article that color is underused, but the state of the art has moved way past what the author’s tools are currently configured to provide.
Jetbtains IDEs let you configure this - my favorite use is to highlight kotlin extension functions differently than normal functions.
This kind of highlighting as a secondary information channel for compiler feedback is great. Color, weight, italics, underlines - all help increase information density when reading code.
Reminds me of my college note taking practice. I used to take notes with about 3-5 different color pens, and my friends used to be puzzled about it (it sure looks weird to swap between pens often).
I used to reply that the color pens made it easier to keep context such as what teacher said was important, what I found difficult, when in the note I had an "aha!" moment, side comment from me, Q&A asked by student during lecture, or how certain things written down now is related to the point made earlier/later in the lecture/notes.
Text (note) is the content but our (at least mine) attention are not really made for plain text. There's so much more you can play with visual information.
Tangential but you are the exact target audience for those pens that were popular in the 2000s, where it was 4 colored ball points combined into one pen.
I still remember my math undergrad notes from years ago: definitions had a prefix Def circled red, theorems -- Th in blue, examples -- Ex in green. Made it much easier to review my notes before midterms / exams.
Whenever I jump into excel or any spreadsheet app, light light blue is user input and light light red / pink / salmon is final formula output. Makes it so much easier.
I'd like to have syntax highlighting that changes based on what I have selected on the IDE. If it's a function, highlight everywhere it's used, make their arguments have different colors, etc.
Once seen, this is so obviously missing. Why aren't we doing this?
I've used the Kate editor for years, it has a short list of strings that are auto-hilited... and I use frequently. If only I could edit and to that list ... wherever it's located!
If only there were a way I could highlite -one- string, and then use a single key to move from that instance to the next!
We are doing it. IntelliJ has done it for years. To highlight a string and move from one instance to the next select it, Edit -> Find Usages -> Highlight usages in file then use Next Highlighted Usage.
Other things it can show you via highlighting:
1. Bugs. The online static analysis will highlight code likely to be in error.
2. Dead code. It's rendered in grey.
3. Code that won't execute in this debugger session. Same.
4. Identifiers you chose to temporarily highlight.
5. Mutable vs immutable variables. Also: mutable variables that are never actually mutated.
6. The assertion that failed in the last unit test run.
You can also create your own smart highlighters using semantic search (it's sort of a grep for ASTs).
And a gazillion more. People still using plain programmer text editors are missing out on a lot of features.
The real magic trick is that I used S-expression to pull up all the first-encounter/deeply-nested keywords touched to its Vim syntax 'nextgroup=', and region block-offs.
Basically said, I complied complex EBNF into Vimscript zeal and need for pure-deterministic LL(1) syntax tree. (Vim regex is weird, must order by largest static pattern first to most wildly wildcard pattern lastly within single regex string).
I have tried it before, unfortunately never found it that handy. For starters you need a pretty complicated expression not already disambiguated by layout to start to care. Which is the sort of thing people try to avoid anyways. Second, there are not a huge number of colors that are super distinct when not right up against each other, doubly so when you consider all of them must be very distinct from the background color. Lastly there's a lot of finickiness with other usages of colours conflicting.
I would actually prefer the opposite. Render the () characters as different matching glyph pairs. The space for distinctive asymmetric glyphs is a lot larger and not generally very loaded because people code 99.9% in ascii.
The built-in editor of Casio calculators is doing this, on some crappy 80x8 screen. I think this is not too uncommon. My editor only colours the parentheses that I currently edit, which I find much less intrusive and has the same (I would say a better) effect.
Terrible idea for color-blind people. And I don't even mean severely color-blind, like when you can't distiguish red from green at all, but something more common and less severe, like deuteranomaly - where it's shades of these colors that are hard to distinguish.
The color blind would still see the parenthesis at least like they see them currently without the color coding though, given they are distinguishable from the background by lightness.
Those people are able to distinguish suggested colors - the example is using primary colors. And at worst, it would be the same as all parenthesis being the same color (black).
no, its not. first of all either I sense brightness differently, or I use it to compensate. so highlighting of any kind make the text a mix of brightnesses from dingy to glaring across the text.
since it take me effort to actually parse the colors, this is a constant distraction.
so I can read monochrome text just fine, but multi-colored text really slows me down.
I thought this was going to go into an extended ascii, maybe 2 bytes per character, 2 bits for rgb, a highlight flag ( as opposed to coloring the words) and an unused bit in honour of ascii.
That way the colour can be defined at write time, languages don't need to implement them, they can be like whitespace and you can use color for whatever you want. Colour would be ignored by both compile and runtime of course... unless it wasn't
I like using syntax highlighting when it breaks. For example, if all code below a particular line is a single color, then I probably forgot an end-quote or something. But this has become less uniquely useful due to the broader integration of parser-driven linters (e.g. tree-sitter), which—besides being able to drive highlighting—can explicitly deliver inline hints or parsing errors.
All that said, I'm one who appreciates information density! How about coloring branching code paths/call stacks?
My keyboard has a concept of "layers," which allows each key to map differently depending on the layer. I've seen this used to make a numpad or to have a QWERTY and DVORAK layer. What if highlighting was the same? Instead of competing for priority over the color channels, developers could explicitly swap layers?
> All that said, I'm one who appreciates information density! How about coloring branching code paths/call stacks? > > My keyboard has a concept of "layers," which allows each key to map differently depending on the layer. I've seen this used to make a numpad or to have a QWERTY and DVORAK layer. What if highlighting was the same? Instead of competing for priority over the color channels, developers could explicitly swap layers?
I was thinking about coloring logic /scope blocks as a way to help visualize scope and flow, even if it required static analysis and a simple script it could be useful when I need to debug
For the last five years I've been working on this problem!
To solve it we need to be able to describe the structured content of a document without rendering it, and that means we need an embedding language for code documents.
I hope this doesn't sound overly technical: I'm just borrowing ideas from web browsers. I think of my project as being the creation of a DOM for code documents. The DOM serves a similar function. A semantic HTML documents has meaning independent of its rendered presentation and so it can be rendered many ways.
CSTML is my novel embedding language for code. You could think of it like a safe way to hold or serialize an arbitrary parse tree. Like HTML a CSTML document has "inner text" which this case is the source text if the program the parser saw. E.g. a tiny document might be `<Boolean> 'true' </>`. The parser injects node tags into the source text, creating what is essentially the perfect data stream to feed a syntax highlighter. To do the highlighting you print the string content if the document and use the control tags to decide on color. This is actually already how we syntax highlight the output from our own CLI as it happens. We use our streaming parser technology to parse our log output into a CSTML tag stream (in real time) and then we just swap out open and close node tags for ANSI escape codes, print the strings, and send that stream to stdout.
Here's a more complicated document generated from a real parse: https://gist.github.com/conartist6/412920886d52cb3f4fdcb90e3...
Having this in the editor leads to people removing that information from other channels. For example class properties are now often coloured, so people do not name or refer to them to denote that they are not some random other variable. I don't like this, because I think all information should be in the code itself and it can be really annoying once you work and haven't your favorite editor available in the correct setup.
with the same argument i used to reject syntax highlighting in general. all the information should already be in the code. if i can't read that, then the syntax of the language is bad. syntax highlighting would just make me lazy. i have since decided that i like being lazy so i do like to use it, but you make a good point, if the use of colors makes you not write something in the code that you would otherwise, then that's not good. i can see that with syntax highlighting too, where the colors help me read code even if it is badly formatted, so i am less inclined to format for readability.
True, but I haven't experienced that for general syntax highlighting. I do read the code through cat or in other contexts often enough that I haven't lost the ability to read code at normal speed without syntax highlighting. Do you have an example?
i wasn't talking about the ability to read code, even though that was my initial argument for rejecting colors. what i mean is that without colors i might take more effort to write/format code so that it is more readable, instead of relying on syntax highlighting:
with colors this is perfectly readable, because if, for and return appear in red, and other keywords in blue. so they stand out, making the structure more visible than without colors.
without colors i might prefer to write something like this. using braces around each block, and line breaks, to make each part stand out. without colors clearly the second is easier to read.github uses a different color scheme but maybe you can get the idea:
https://github.com/pikelang/Pike/blob/fe4b7ef78cc26316e62e79...
I see. I'm typically quite pedantic and start reformatting code manually as soon as a single space is off.
I've never seen that language. Looks C like (e.g. sizeof), but seams to have a harder type system.
> Color carries a huge amount of information. Color draws our attention. Color distinguishes things. And we just use it to distinguish syntax.
I didn't actually find the uncoloured circle to be that much harder to spot.
Before looking at the concrete examples, I thought:
> Understanding the syntax similarly isn't hard scanning over the code. But if the words were coloured in a way that didn't correspond to syntax, I expect I would find it distracting, in the same way as those experiments with colour words written in a different colour from what the word indicates. Trying to make use of that second information channel isn't necessarily a good idea.
> I like the aesthetic of syntax-coloured code. But also, it's reassuring. I'm not using it to help understand the syntax, but to confirm that there's software in place that understands the syntax the same way I do.
After: the rainbow parentheses honestly don't help that much, not for a language like LISP anyway. The context highlighting seems to work much better. But then, as I go on through the examples... these are all really doing the same kind of thing that syntax highlighting does! They're about the structure of the code. And I'd have to shift my expectation, but once aligned, it's again something I'd perceive the same way — as something to confirm my understanding rather than eliciting that understanding.
Perhaps being able to switch between colouring modes would change that, but I don't know how quickly I could get used to that technique. And then, the kind of code where most of these things would help is smelly anyway. I already try to avoid this kind of nesting. Maybe import and argument highlighting (which could be used at the same time, along with highlighting for class attributes and closures in Python?), though...
Many (most?) of these are achievable with neovim and tree-sitter (a plugin that gives you access to the AST), and surely many other editors. I have plugins installed right now that do several of the things that are mocked up here. Many more are done with virtual text and not color, but I don’t see why you couldn’t use highlighting instead.
I agree with the broader point of the article that color is underused, but the state of the art has moved way past what the author’s tools are currently configured to provide.
To be fair, the article is over 5 years old.
The author seemed to be unfamiliar with tree-sitter (first appeared in 2018) and incorrectly assumed Atom used TextMate.
Since then it's gotten much more popular and adopted by other editors.
Good catch, thanks!
Jetbtains IDEs let you configure this - my favorite use is to highlight kotlin extension functions differently than normal functions.
This kind of highlighting as a secondary information channel for compiler feedback is great. Color, weight, italics, underlines - all help increase information density when reading code.
Reminds me of my college note taking practice. I used to take notes with about 3-5 different color pens, and my friends used to be puzzled about it (it sure looks weird to swap between pens often).
I used to reply that the color pens made it easier to keep context such as what teacher said was important, what I found difficult, when in the note I had an "aha!" moment, side comment from me, Q&A asked by student during lecture, or how certain things written down now is related to the point made earlier/later in the lecture/notes.
Text (note) is the content but our (at least mine) attention are not really made for plain text. There's so much more you can play with visual information.
Tangential but you are the exact target audience for those pens that were popular in the 2000s, where it was 4 colored ball points combined into one pen.
I still remember my math undergrad notes from years ago: definitions had a prefix Def circled red, theorems -- Th in blue, examples -- Ex in green. Made it much easier to review my notes before midterms / exams.
Whenever I jump into excel or any spreadsheet app, light light blue is user input and light light red / pink / salmon is final formula output. Makes it so much easier.
Reminded me of ColorForth. Ist is a Forth dialect where color also carries code information
I'd like to have syntax highlighting that changes based on what I have selected on the IDE. If it's a function, highlight everywhere it's used, make their arguments have different colors, etc.
Once seen, this is so obviously missing. Why aren't we doing this?
I've used the Kate editor for years, it has a short list of strings that are auto-hilited... and I use frequently. If only I could edit and to that list ... wherever it's located!
If only there were a way I could highlite -one- string, and then use a single key to move from that instance to the next!
We are doing it. IntelliJ has done it for years. To highlight a string and move from one instance to the next select it, Edit -> Find Usages -> Highlight usages in file then use Next Highlighted Usage.
Other things it can show you via highlighting:
1. Bugs. The online static analysis will highlight code likely to be in error.
2. Dead code. It's rendered in grey.
3. Code that won't execute in this debugger session. Same.
4. Identifiers you chose to temporarily highlight.
5. Mutable vs immutable variables. Also: mutable variables that are never actually mutated.
6. The assertion that failed in the last unit test run.
You can also create your own smart highlighters using semantic search (it's sort of a grep for ASTs).
And a gazillion more. People still using plain programmer text editors are missing out on a lot of features.
In Vim: * to highlight a word, n/C-n to move between them. You need to have the hlsearch option set.
Ive completed what is arguably the largest syntax highlighting: nftables v1.1.4 script file (and command script) for Vim/NeoVim.
Before that, I completed arguably the 2nd largest syntax highlighting: ISC Bind9 (most versions)
https://github.com/egberts/vim-syntax-nftables
https://github.com/egberts/vim-syntax-bind-named
My secret weapon was using a smaller highlighted syntax to project even faster completion of these larger syntax tree: EBNF
http://github.com/egberts/vim-syntax-ebnf
The real magic trick is that I used S-expression to pull up all the first-encounter/deeply-nested keywords touched to its Vim syntax 'nextgroup=', and region block-offs.
Basically said, I complied complex EBNF into Vimscript zeal and need for pure-deterministic LL(1) syntax tree. (Vim regex is weird, must order by largest static pattern first to most wildly wildcard pattern lastly within single regex string).
Rainbow nest braces, command/statement/keyword/unit/integer coloring.
For my next trick, I need to determine which route to go next (maybe HN can help me here).
- JetBrain's properitary LSP
- VSCode textmate LSP?
- treesitter
- or something more LSP mainstream, if any.
Kinda disappointed that there is no holy grail for both syntax-highlighting and autocompletion.
Was looking forward to adding hint-hover as well.
Rainbow parentheses blew my mind. Why isn't every editor already doing this?
I have tried it before, unfortunately never found it that handy. For starters you need a pretty complicated expression not already disambiguated by layout to start to care. Which is the sort of thing people try to avoid anyways. Second, there are not a huge number of colors that are super distinct when not right up against each other, doubly so when you consider all of them must be very distinct from the background color. Lastly there's a lot of finickiness with other usages of colours conflicting.
I would actually prefer the opposite. Render the () characters as different matching glyph pairs. The space for distinctive asymmetric glyphs is a lot larger and not generally very loaded because people code 99.9% in ascii.
The built-in editor of Casio calculators is doing this, on some crappy 80x8 screen. I think this is not too uncommon. My editor only colours the parentheses that I currently edit, which I find much less intrusive and has the same (I would say a better) effect.
most have an extension or setting to do this already!
Terrible idea for color-blind people. And I don't even mean severely color-blind, like when you can't distiguish red from green at all, but something more common and less severe, like deuteranomaly - where it's shades of these colors that are hard to distinguish.
The color blind would still see the parenthesis at least like they see them currently without the color coding though, given they are distinguishable from the background by lightness.
Those people are able to distinguish suggested colors - the example is using primary colors. And at worst, it would be the same as all parenthesis being the same color (black).
no, its not. first of all either I sense brightness differently, or I use it to compensate. so highlighting of any kind make the text a mix of brightnesses from dingy to glaring across the text.
since it take me effort to actually parse the colors, this is a constant distraction.
so I can read monochrome text just fine, but multi-colored text really slows me down.
I thought this was going to go into an extended ascii, maybe 2 bytes per character, 2 bits for rgb, a highlight flag ( as opposed to coloring the words) and an unused bit in honour of ascii.
That way the colour can be defined at write time, languages don't need to implement them, they can be like whitespace and you can use color for whatever you want. Colour would be ignored by both compile and runtime of course... unless it wasn't