De-duplicating EndNote results against a previous search

Well hello my friends! It’s been a long time since I’ve posted. Forgive me, as much has happened in the last year, including: moving across an ocean, subsequently moving across a city, starting a busy freelance information specialist business, and many mundane crises, trips, and side-projects in between.

I was tasked recently with updating a systematic review search with a new and improved search strategy completely unlike the previous one. There were new search terms added to this search, but also several irrelevant search terms that had been deleted. I’d done systematic review updates before, but I’d always simply used date filters in the databases to capture results from the date of previous search.

But this time, the researchers wanted any articles that would be captured by the new search, in any date period, and wanted to ensure they weren’t screening any citations that had already been previously screened with the old search.

I’d heard other information specialists talk about using EndNote to de-duplicate against a previous search, but never tried it myself. It just seemed unduly complicated. Date filters seemed to be working perfectly fine.

It was around this same time that I found out that date filters were not perfectly fine.

One day, I went to date limit an Embase search using the advice from a LibGuide at a high-ranking university…. and was horrified to find out that a not insignificant number of citations that ought to have been picked up had not.

Cue a minor panic as I tried to figure out whether I had royally screwed up any of my previous projects.

Friends, I have learned the error of my ways, and will henceforth de-duplicate my systematic review update searches in EndNote when possible. Cross my heart, etc, etc.

As usual when picking up a new skill, I went to Twitter to see what all the experts were doing.

Here follows the method that I ended up using. I’ve documented it for my own purposes and hope that it can come in handy for others as well.

1. De-duplicate your total search results in EndNote, as normal.

You can use whatever process works best for you. I tend to use a modified version of Bramer et al, 2016 in which I progressively choose different field combinations in EndNote to test for duplicates, and manually go through the results. The field combinations suggested in the article include (in this order):

  • Author | Year | Title | Secondary Title (Journal)
  • Author | Year | Title | Pages
  • Title | Volume | Pages
  • Author | Volume | Pages
  • Year | Volume | Issue | Pages
  • Title
  • Author | Year

But if you want to get fancy about it, the article supplies a more complicated process than this.

2. Label your citations by search date.

At this point, you’ll want to load up your citations from the previously conducted search into a separate EndNote Library. Then, use one of the custom fields to mark these citations.

  1. Select one of your citations from the “All Refs” group, then click cmd + A or ctrl + A to select all in the entire library.
  2. Then, go to tools, then Change/move/copy fields
  3. Select “Custom 1” (or another field of your choosing), then “replace whole field with”
  4. Choose text that is meaningful for you for remembering which citations these are. Something as simple as “OLD” may suffice (this is what I did, based on a Twitter tip).
  5. Next, do the same for your “new” search results.

Note that this is an important step if your search strategy has changed such that some results that were previously returned will not be returned in the new search. Otherwise, you will end up re-screening those articles!

3. Combine your “old” and “new” EndNote Libraries together

To combine your two libraries together, navigate to your “old” library and select all the citations by clicking ctrl + A or cmd + A.

Then click “references”, then “copy references to”, and choose your “new” library. Easy peasy!

4. Remove duplicates

Use your EndNote Library which contains both your “old” and “new” records (both of which have previously had duplicates removed!), and remove your duplicates as you normally do, or following the process in Step 1.

But this time, there’s one big difference – every time you find a duplicate, instead of removing the duplicate record, you’ll remove BOTH records, since they represent a previously screened record that you won’t need to screen again.

5. Remove any remaining previously screened citations

This step won’t be necessary if no search terms have been removed since the original search was conducted.

In my case, the search had changed drastically since its creation by someone else, and I needed to remove any citations that were picked up by the original search, but not by the new one. Removing these records is easy if you have followed Step 2, above!

Simply create a smart group by right-hand clicking over “my groups” in EndNote. Then, set the parameters to find your old citations (e.g. “custom 1”, “is”, “OLD”). Then, navigate to your smart group and delete all the citations in this group. These have already been previously screened and weren’t retrieved by the new search.

And that’s basically it! I was able to tackle this new skill that originally seemed kind of hard, and you can too!

For more information, the following papers may also be useful:

Bramer WM, Bain P. Updating search strategies for systematic reviews using EndNote. Journal of the Medical Library Association: JMLA. 2017 Jul;105(3):285.

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association: JMLA. 2016 Jul;104(3):240.

Advertisements

3 thoughts on “De-duplicating EndNote results against a previous search

  1. I use a bit of a different method:
    1. Open old EndNote library. My libraries typically have groups based on databases – ie. A Medline, B Embase, etc
    2. Add new search following the same model but flagged as update – ie. C Medline update, D Embase update, etc. This means they appear lower down the list of groups than the original search and have higher record numbers.
    3. Deduplicate as usual, removing the item with the higher record number when duplicates exist.
    4. Export only the “update” groups to share with the researchers for screening

    Like

    • Thanks for sharing your method – and great ideas! I will test this out sometime and see how it works for me.

      I do always add my citations to groups based on database as well – but have always backed up the library with all citations and groups and then removed the groups prior to de-duplication. Otherwise I think some researchers might get confused thinking those were the total citations retrieved from each database in the de-duped library. I will rethink this strategy now!

      One of the reasons that I created this post and reached out to Twitter for support was that I could see there were many creative ways of accomplishing the same tasks. And I’m not totally convinced my way above is the best way! 🙂

      Like

  2. Thanks for writing up the method. I now have another one to do (this time it is not an update search, but the proper search following a pilot screening exercise). Will follow your steps and report back. This blog is such a good reference source.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s