Where's the IDE (Integrated Development Environment) for personal information?

I write down everything I want to remember. That way, instead of spending a lot of time trying to remember what it is I wrote down, I spend the time looking for the paper I wrote it down on. -- Beryl Pfizer

I've dreamed up an information-managing tool that should exist, and it's driving me nuts working without it. Here are a few initial notes; comments are very welcome.

In training myself to be a productivity consultant, my information tracking and capturing needs have gone through the roof. For example, I need to manage things like:

• networking information (who, when, how I connected to them),
  
• blog ideas,
  
• useful productivity tips for myself and clients,
  
• marketing and web site ideas,
  
• notes from books I read [2],
  
• notes/ideas for projects,
  
• lessons learned and personal challenges,
  
• inspirational reading,
  
• contact information,
  
• project hours (for example, in a coaching exchange I'm doing with someone),
  
• ...

(Note that I don't use this file for my GTD implementation; for that I use a paper planner with sections for my calendar, Next Actions, projects, etc. [3])

As I've described before [4], I use a plain text file with simple formatting to break it into individual records, and WikiCase (AKA CamelCase or WikiWords) to help navigate and find needed information. Here's an excerpt:

  ----
  IDEA: IdeaMatt: from JasonWomack: you teach what you most need to
  learn. in my case: ...
  (2005-11-15 21:40:42)
  ----
  IDEA: IdeaMatt: the irony of trying to set something up (take action)
  with people who need help taking action - slow email responses,
  unreturned phone calls, etc.
  (2005-11-15 14:38:14)
  ----
  IDEA: IdeaMatt: idea of copying todo lists when reach 1/2 full
  from "Time management for dummies"
  (2005-11-15 13:40:49)
  ----
  ...
  ----

After using this scheme for over a year, I've realized the tool is ready to grow up, and what it wants to be is something akin to a programmer's tool for information. Now I'm the first to admit that I'm an information "power user," and that a tool that works well for me would probably make most people run screaming with tears streaming down their faces, but hey - my readers are smarties!

What I've in mind is something like the next level past Wikis (think Jot meets Dabble DB outside the mad doctor's castle, and gets attached by half-crazed Lotus Agenda townspeople carrying pitchforks and torches, runs smack into Chandler, and gets stabbed by Planner Modefor good measure). It would work like an information IDE, something like the stellar IntelliJ IDEA, but for semi-structured data.

Here are some of the features, with programming comparisons to IDEA:

• Views: The UI would have multiple views, including a hierarchical one like wikidPad, a graphical one like MindManager, and a chronological one like Lifestreams (the one my text file tries to emulate).
  
• Editing: Smart completion of WikiWords, time-stamping of entries, etc. NB: Completion is crucial for efficiency, and for consistent use of tags.
  
• Navigation: Basic link following, including Find Callers and Jump To Definition, bookmarks, and forward/back functionality. (Note: the graph view may be better for higher-level navigation and abstraction.)
  
• Reports: One thing I'd love to have is a histogram showing who I've talked to (WikiWords, more generally) on a timeline so that I can see who I've not had contact with in a while, what kinds of entries I've been focusing on, etc.
  
• Refactoring: Often I'll find that I've duplicated some concept (such as a blog idea or a project), and I'll want to do the equivalent of XP's refactoring, including Extract Entry, Create Link, Inline entry, Rename Entry, etc.

That's the initial thought. However, there's something deeper that goes beyond these relatively straightforward operations. On top of them, I see the need for what I've been calling "self-structuring" operations, ones that support emergent organization of the data, esp. for records with more traditional database-like structure [5]. More on this later, but it would move the idea from a tool to a framework for information applications, enabling it to specialize itself (with guidance) to tasks like finances, expenses, contacts, and what I call "semantic bookmarks."


Related Reading

For additional thoughts on this, interested readers might enjoy the following drafts:

• CRAM - a Contextual Relational Augmented Memory for Personal Data Mining
  
• An intelligent email system
  
• Ideas on a next-generation personal information manager

Note that they refer to our research project Proximity. It's a system supporting relational knowledge discovery (advanced data mining), but my original interest was as a framework for implementing some of the above ideas.


References

• [2] You might enjoy How to read a lot of books in a short time.
  
• [3] My current GTD setup is described in Fare thee well Hipster PDA - I barely knew ye.
  
• [4] You can read more about the text file format in My Big-Arse Text File - a Poor Man's Wiki+Blog+PIM, Pickle jars, text files, and creative idea capture, and Photo Blogs, Wikis, and Memories for Life.
  
• [5] My friend David Phillips also talks about this in Gems in a treasure chest.

Comments

[Chris]

So here's my first question: why would you need one? What itch are you scratching with this IDE? Are you trying to come up with a way to then develop to APIs for other people's ideas?

[Matthew Cornell]

Chris: The itch? The simpler version: A tool that lets me do my stuff quickly. The meta-version: A framework for creating tools to let people do their (information-handling) stuff quickly. Thanks for asking!

[Joe McCarthy]

Um, at the risk of suggesting the obvious, have you considered blending your passion for GTD with your technical skills and experience in the education realm all together, and actually building this tool (and offering consulting services to people and organizations who want to use information more effectively to get things done)?

[Anonymous]

I think an IDE for information is a great idea. I find myself bouncing around from info tool to info tool because none of them are really powerful and useable enough for a large, dynamic set of information.

rgable

[Matthew Cornell]

...blending your passion for GTD with your technical skills and experience in the education realm all together, and actually building this tool

Thanks very much for the suggestion/question, Joe. I took this track, lost enthusiasm, and decided I'm burnt on technology. I'm now 1/2 time at work, and it's possible I'll get some motivation back, but without some major external event (e.g., Microsoft says they have to have me to lead building it), I won't be doing it.

Thanks!

[Matthew Cornell]

I think an IDE for information is a great idea. I find myself bouncing around from info tool to info tool because none of them are really powerful and useable enough for a large, dynamic set of information.

Thanks for the comment, rgable. You've definitely got it.

[David]

Have you checked out [ Tinderbox | http://www.eastgate.com/Tinderbox/ ]
I don't have the money to buy it but it sounds like a fun thing to try. It doesn't cover all the bases but seems to come closer to being truly all-encompassing in terms of different types of information in different contexts.

[Matthew Cornell]

David: Thanks for the pointer to Tinderbox. I had seen it a year or two ago and thought then (as now) that it's very cool. I also see they have a Windows version coming soon. I like that it's a graphical system with user-controllable attributes - neat!

[Anonymous]

Tikiwiki will do all this for you - even the visualization

Otherwise check out the touchgraph wiki browser that straps onto most things

[Matthew Cornell]

Anonymous: Thanks for the TikiWiki pointer - I'll definitely check it out, and for the Touchgraph UI - I've played with it before (I only know about it because we have a visualization component in our lab's software, and finding/writing a great graphing engine is a real challenge)

[Anonymous]

Have you taken a look at [ Intertwingle | http://www.mozilla.org/blue-sky/misc/199805/intertwingle.html ] ?

It's more an idea than actual software, but when I stumbled over it a couple of months ago, I thought to myself "wouldn't it be cool if this thing were 'smart'?"

Some kind of agent to crawl all over structured data, remind me of stuff when I need to remember it.. automagically without me having to enter a remind(1) entry or anything.

Anyway. Thanks for the inspiration and whatnot... I hope that Intertwingle makes its way into one of your linkdumps.

[Matthew Cornell]

Hi Anonymous. Yes, I think Intertwingle is very cool, and I like your ideas on making it smart, and doing automated reminding - neat. Thanks!

[Reg Cheramy]

Zigtag will be a lot of what you're looking for.

http://www.zigtag.com/

[Matthew Cornell]

Thanks for the teaser, Reg. Please let me know if you want to talk about it.

[Elizaveta Revyakina]

Hello Matthew,

Perhaps it may surprise you but JetBrains has developed a toll which will likely suit most of your requirements to the personal information environment. The name of this tool is Omea Pro (http://www.jetbrains.com/omea/)

In two words, Omea Pro allows to manage several types of information sources: e-mail, tasks, notes, web bookmarks, clippings (excerpts from web pages and other stuff), instant message conversations, newsgroups, rss feeds, and contacts.

Omea Pro has a number of quite unique features such as categorizing your stuff, possibility to filter the information using different views, and many more.

Here is Omea's Tips'n'Tricks blog which contains a number of tips inspired by Omea users: http://blogs.jetbrains.com/omea/

You can download the latest version (2.1.4) at http://www.jetbrains.com/omea/download/download.html

[Matthew Cornell]

Thank you for the comment, Elizaveta. I looked at Omea around 2004-08, so I suspect it has improved a lot. At the time I found it very intriguing, but a bit too communication-oriented, which meant - for example - that simply creating an object was not possible. It had to be a contact, etc. I suspect it deserves an updated look.

Thanks for reading!

[Charles]

You should investigate orgmode in Emacs. It does many
of the things you want but storing everything in
plain text files. Go to http://orgmode.org for details.

I discovered org-mode by first reading about your big plain text
file approach, then explored Planner and finally discovered
org-mode.

Remember mode is a great way to capture thoughts on the
fly, and you can create templates - useful for contact
information, book ideas.

Clock mode is useful for recording time spent against
projects.

[matthewcornell]

Thanks for the tip, Charles. Clock mode is new to me! I haven't tried these modes because 1) my current setup is working pretty well, and 2) I'm not an Emacs whiz, so installing seems a big daunting.

Others reading might appreciate the Emacs add-on suggestions readers offered in [ My Big-Arse Text File - A Poor Man's Wiki+Blog+PIM | http://matthewcornell.org/blog/2005/08/my-big-arse-text-file-poor-mans.html ], including org-mode planner-mode and remember-mode.

Thanks for the comment.