Mediums to try out algorithms?
By definition algorithms are independent from the medium they run on. E.g. I use Excel tables to play with data structures and to do some move/shuffling/marking experiments before implementing an algorithm in a programming language.
What tools and techniques do you use to design and simulate the function of an algorithm? How do you use drawing programs? A special specification language?
Algorithms are independent of language, true. But any medium you use is a language, period. Using Excel means you're using the excel "language" (rows, columns, cells, etc.) to express some of your algorithm. Maybe not the complete, finished product, but you've expressed it in Excel.
UML diagrams are a language for expressing an algorithm. A sketch on paper is a language for expression. It's hard to separate the platonic ideal algorithm from all concrete representations. No matter what you do, you're expressing it in some language.
The trick is to separate the algorithm from specific language features and limitations.
Any "informal" notation will help do this. English (or other natural language), mathematics, diagrams, etc., are all candidates for expressing an algorithm in a language that's free from implementation quirks and problems.
I start with an overview in English. Plain text, not even MS-Word or something where formatting is a distractor.
For really complex things, a little supplemental mathematics helps put formal assertions around the state of the program.
Also, UML diagrams help. I use Argo UML -- cheap and effective.
For more on this topic, you want to read about formal verification systems.
I usually sketch it out on paper, then try a quick implementation in Common Lisp and test it on the REPL.