Chef's search feature is powered by Lucene via Solr. Every time an object is created or updated in Chef (e.g. a node, a role, or a data bag) the object is converted to JSON and put on a queue (we use RabbitMQ) where it will be picked up for indexing. The indexer expands the JSON into a set of key/value pairs (deep structure is flattened by concatenating keys with underscore) and sends the data to Solr for indexing.

The search outage was the result of Lucene doing a merge of its index that took minutes. Analysis of our Lucene index revealed that we had well over 900,000 fields in our index. If you look hard, you can find benchmarks on Lucene using 30 fields. Our system was using Lucene wrong.

In addition to mapping each flattened key into a field, the indexer was "expanding" all compound keys by inserting an 'X' to allow a kind of wild card searching. For example, the JSON:

{"a": {"b": {"c": 1}}}

would have a flattened key of a_b_c which would get expanded to:

X_b_c
a_X_c
a_b_X

This was a non-optimal approach to say the least -- yet Lucene actually made it work for quite some time.

We needed to change how Chef objects were indexed to avoid creating an unbounded number of fields in the index. The solution we came up with was to concatenate the flattened key value pairs with a delimiter (we chose __=__) and put all of the concatenated pairs into a single content field in the search index separated by whitespace. Besides reducing the number of fields in the index, this approach eliminated the need for the "X" expansion since Lucene wild card queries using "*" could be used instead; this saved considerable time and memory during the indexing preprocessing phase.

Experiments with simulated node data were run to validate the new approach. Two 10,000 node save test scenarios were run: the first was optimistic, in which two random keys were added to a node being saved, and the second pessimistic, in which 50 random keys were added for each save. The results showed a decrease in average commit time of 4x and 23x, respectively.

This left us with two problems. We needed to reindex our data and all queries would fail once we did; querying the new index required a new query syntax.

The reindexing was solved with a well practiced deployment plan and a small amount of scheduled maintenance. Dealing with the change in the query syntax was somewhat more challenging.

Chef's query syntax directly exposes the Lucene syntax and, as a result, the initial index layout. Chef object keys map to fields in the query. For example, to search for all nodes with the "web" role, the user query would be role:web which will be passed along to Lucene unchanged. Lucene will interpret this as a search for documents with "web" in the "role" field. The equivalent query for the new "no-fields" index looks like this: content:role__=__web. In order to deploy the new indexing scheme, we needed to be able to convert user queries into the new format.

We were under operational duress at the time and needed a fast-as-possible solution. By examining system logs, we determined that we could cover the types of queries actually in use with a few regular expressions. If you're ever looking for a definition of "quick and dirty", this is it. To help verify the solution, we wrote a test harness that populated an organization with a known set of objects and randomly generated queries with Boolean operators in such a way that the result set was computable. This helped increase our confidence in our duct tape solution.

Having introduced a series of regular expressions into our query handling code, we now had an additional problem. The regular expression based solution was a stop-gap measure and didn't support many aspects of the Lucene query syntax. For example, we failed to handle the ! and - operators (see Lucene syntax). The query -role:blah was transformed into content:-role__=__blah when it should have been -content:role__=__blah. We considered adding more regular expression based replacement rules to the already somewhat convoluted logic, but it became clear that we'd need a fair bit more logic to be able to identify the leading dash and ignore the embedded dash in a query like tags:b||(-role:web-server tags:a) (which should be transformed to content:tags__=__b || (-content:role__=__web-server content:tags__=__a).

In order to restore support for more of Lucene's flexible syntax and to end up with a maintainable query handling system that isn't a laundry list of regular expressions, we began work on code to parse Lucene's query syntax. The initial implementation used a Parsing Expression Grammar (PEG) created with Treetop. Here's the grammar file we used. The result was a query handling system that we have full control over and that provides a layer of abstraction over Lucene's query syntax. A side-benefit of the full-parse approach is that we can now validate user queries much earlier.

Since deploying the reorganized index, Lucene's commit times have remained short and consistent even in the face of significant growth of the index. We've ported the Hosted Chef search API endpoint to Erlang and ported the Lucene PEG to Erlang using neotoma. We have been able to address a few further edge cases of the Lucene query syntax in a structured and robust fashion (no regular expressions required).

Our community has grown to over 15,000 users, and there are over 500 developers and 100 organizations contributing to and expanding Chef.

In short… we’re ready for our own conference!

In a few days we’ll be officially announcing #ChefConf 2012, our inaugural community conference.  #ChefConf will be held May 15-17th in Burlingame, California.  We’re really excited about what we’re putting together.

Submit your session ideas for the
#ChefConf 2012 Call for Proposals now!

Opscode is proud to be a sponsor of the Southern California Linux Expo (SCALE) this upcoming weekend in Los Angeles. SCALE is one of the premier open source events in the United States and we encourage you to attend. We’ll be participating in a number of events, giving a presentation and we’ll have a booth on the Expo floor.

Chef Events

Friday January 20

Build a Cloud Day: How to build and manage cloud computing environments using free and open source software. There will be an “Automating Your Cloud with Opscode Chef” session presented by Opscode employee Matt Ray.

Look for Opscode and Chef folks at the DevOps Day LA and at the OpenStack Happy Hour!

Saturday January 21

Visit the Opscode booth at the Expo Saturday and Sunday (#53).

Catch up with other members of the Chef community and grab a drink at the Chef Birds of a Feather session (Los Angeles BC room).

Sunday January 22

Infrastructure Automation with Chef: Opscode’s Sean OMeara will be giving an introductory Chef talk to help people get up to speed.

We would like to thank the Southern California Linux Expo for all their support and encourage you to come out and see us this weekend!

Food Fight is a new podcast for the Chef community, discussing and summarizing the assorted news and topics of the day. It’s hosted by community member Bryan Berry and Opscode employee Matt Ray. They plan on interviewing lots of people from the greater Chef community and publishing every few weeks.

The first episode is up now, where they talk about Chef in 2012 with guests Eric Wolfe and Jay Feldblum. The homepage is http://www.foodfightshow.org and the show is also available on iTunes for subscription.

• Opscode’s community site grew to over 11,000 registered users and 380 community cookbooks supporting everything from Apache to Zabbix.
• Over 500 people and 100 organizations are contributing to Chef, with release MVPs.
• We ran our very first Opscode Community Summit which was filled to the max with over 110 people passionate about the future of Opscode & Chef. It was quite moving to have so many people come together who cared about Chef so much!
• We’ve had an amazing time teaching people about Chef in our training programs, and will have major announcements in the year to come as the training expands.

The community site has received some long-awaited love this year, most notably with the addition of a source code browser. We also had our first cookbook contest which showed a dedicated and creative set of users out there solving interesting engineering problems with Chef.

This year saw the release of Chef 0.10. This release of Chef introduced the concept of environments, making it much easier to maintain configuration separation between production and other environments while supporting and encouraging other environments being produced by the same process. Paired with devops culture, this helps reduce issues with a different environment when software passes from development and testing into production infrastructure.

Chef 0.10 also introduced encrypted data bags and knife plugins, among other things, providing for increased utility, flexibility and functionality. The Chef Community has fully embraced plugins, actively creating Community Plugins for chef, ohai and knife – each a specialized delivery of particular functionality that can now be repeatably used and conveniently shared. Opscode utilized the plugin feature directly, providing a Knife Windows Plugin for bootstrapping Windows systems using our powerful knife command line tool, in addition to specialized plugins to provide for automated Launching of Cloud Instances with knife.

Opscode continued progress from the significance of 0.10, and are now at Chef 0.10.8: where we gave our fellow Windows Systems Administrators a leg-up in the devops cultural revolution. This release culminates a significant increase of Windows users of Chef this year, which also included our release of an Chef Client Installer for Windows. There are members of the community already deploying this installer using group policy and teaming it with Windows Deployment Services (WDS) to enable bootstrapping bare-metal into fully functional servers. This pattern prevents traditional dusty and broken build documents for meat-cloud built servers as well as golden-images with an uncertain past.

Opscode Hosted Chef experienced organic growth throughout the year. As the user base of Hosted Chef expanded, Opscode responded to the growth by streamlining the management of search indices – resulting in average commit times being cut in half and merge commit times were reduced ten fold. We also upgraded the API Endpoints, and following migration of organizations to the new endpoints, saw standard response times halved from their pre-migration level.

Beyond those engineering efforts to sustain Hosted Chef scalability and reliability, Opscode Hosted Chef customers also benefited from the establishment of Enterprise Support, with Opscode offering an industry-leading Service Level Agreement for cloud infrastructure automation, including 24x7x365 support plans. Hosted Chef customers are receiving solutions to questions with a median response time of 35 minutes and a median resolution time of just over 4 hours for resolution of complex implementation and support issues – and continuous quality improvement is the goal.

Speaking of continuous quality improvement, the Chef Wiki underwent a fairly extensive redesign, restructuring and update throughout 2011. The result is a substantially improved and more useful set of documentation for Chef users, with further enhancements to product and support documentation both planned and in work for the new year.

It hasn’t all been evolutionary growth – there has also been a platform revolution! Opscode unveiled Private Chef for the Enterprise, providing for a highly available, dynamically scalable, fully managed and supported automation environment behind the corporate firewall. We’ve also been honored to have Chef included in both the OpenStack project and in the tools around it like Crowbar in partnership with Dell and Rackspace. We are excited about the future of these projects and honored to be a part of them.

And all of this is just a portion of the fantastic year that 2011 was for Opscode, Chef, and the Chef Community!

Looking back, it is amazing and inspiring the distance that we have come since Chef was introduced and Opscode was founded just a few years ago. And that is nothing compared to what is planned for the future – and what we can all accomplish together.

Thank you to all of our customers and the wonderful open source Chef community for 2011… Welcome to 2012!

I considered searching through nodes for attributes that ought to be unique to these test instances, like the run_list. But, then I remembered that the time of the last Ohai run is stored as an Unix time in the automatic attributes of a Chef node run.

$ date +%s
1324426395
$ date –date="-5 hours" +%s
1324408398
$ knife search node ‘ohai_time:[1324408398 TO 1324426395]‘ -i
2 items found

i-b25c8ad0

i-1e5c8a7c

Above, I generated two time values using the date command, one for the current time and one for five hours ago. Then I used these values to search through my nodes for those who have had a Chef run in the last five hours using a range search with knife. This returned two nodes, which was exactly what I was looking for.

Another time-saving feature of Chef that you didn’t know you even wanted until you find yourself needing it.]]> http://www.opscode.com/blog/2011/12/21/finding-recently-created-hosts-with-chef/feed/ 2