User login
Many are submitted, but few are chosen.
Concerned about the quality of submitted research papers based on large surgical databases that are not accepted for publication, the editorial board of JAMA Surgery has taken the initiative by giving some pointers to would-be authors. The journal editors have published a 10-point checklist of dos and don’ts to address commonly seen problems with submitted manuscripts. In addition, JAMA Surgery collaborated with the Surgical Outcomes Club to commission a series of practical guides on the most widely used data sets in an effort to improve the quality of surgical database research. The Surgical Outcomes Club is a consortium of surgeons and scientists who work to advance health services and outcomes research in surgery that was launched in 2005 at the American College of Surgeons Clinical Congress.
The authors noted that, although JAMA Surgery receives hundreds of submissions of retrospective studies of large surgical databases each year, most of these studies have flaws in the data analysis or use a hypothesis that the data sets cannot address. Hence, the editors do not send most of them out for peer review. “Of those that are sent out for peer review, many are recommended to be rejected by expert peer reviewers as they find major methodological flaws in the use of these otherwise powerful data sets,” the team wrote.
“Research using data sets can be very powerful as the research can address questions and hypotheses using large populations of people. However, the research can have many weaknesses. First, the research is only as good as the data collection for each data set. Second, the investigator needs to be familiar with the types of research questions and hypotheses that can be addressed with each data set. Third, the statistical methodology used to analyze the data is also imperative,” said Dr. Kibbe in an interview.
The checklist begins with a recommendation that the researchers develop a clear, concise hypothesis using established criteria – either FINER (for feasible, interesting, novel, ethical, relevant) or PICO (patient, population, or problem; intervention, prognostic factor, or exposure; comparison or intervention; outcome); the checklist then goes on to include compliance with institutional review board and data use agreements and to emphasize the importance of a clear take-home message that addresses policy or clinical implications.
The series comprises 11 two-page articles that aim to serve as practical guides for using each of the most widely used surgical data sets, starting with the National Inpatient Sample and ending with the Society of Thoracic Surgeons data set. Each article includes a bulleted list of the data set’s attributes, an explanation of its limitations, a history of the data set, an explanation of how the data is collected and what is unique about the set’s features, and statistical considerations researchers should take into account when analyzing the data.
The series concludes with tips from the statistical editors of JAMA Surgery – Amy H. Kaji, MD, PhD, of Harbor–University of California Los Angeles Medical Center in Torrance; Alfred W. Rademaker, PhD, of Northwestern University, Chicago; and Terry Hyslop, PhD, of Duke University, Durham, N.C. – for performing statistical analysis of large data sets: “With bigger data, random signals may denote statistical significance, and precision may be incorrectly inferred because of narrow confidence intervals,” the statistical editors noted. “While many principles apply to all studies, the importance of these methodological issues is amplified in large, complex data sets.”
However, they noted that large data sets are prone to bias and measurement errors. “It is important to respect and acknowledge the limitations of the data,” the statistical team wrote. They also reprise the introductory editorial’s call for a clear hypothesis and take-home message. “The challenge with Big Data is that it requires a carefully thought-out research question and a transparent analytic strategy,” the statistical editors said.
And there are pitfalls. Dr. Bilimoria noted, “We shouldn’t let the database define our research. We should instead be asking interesting questions and then seeking out a database that fits best to answer the question.” He said one particular problem that comes up often for reviewers is trying to discern how researchers arrived at the population of interest in a study. “A lot of inclusion and exclusions criteria are applied, and unless [the reviewer] can see the decisions that were made in the process, some fairly important biases can be introduced unintentionally. We as reviewers would like to be able to follow that exclusion pathway.”
He said, “A problem we frequently see is that these databases change the variable definitions over time – in fact, change the variables over time. So if researchers aren’t checking to see if that variable was reported the same every year of the study and in the same way, they will get spurious results. Similarly, the number of hospitals reporting is important as well since hospitals come in and out of these data sets.”
In their introductory editorial, the JAMA Surgery team noted that the checklist, practical guides, and statistical tips are a three-pronged approach that authors should consult before submitting their manuscripts. “We hope that by following these simple guides, authors can benefit from the collective wisdom of so many colleagues who have successfully completed similar analyses in the past,” they wrote.
Dr. Bilimoria sees great strengths in database research, such as giving researchers a population-level view of how care is being delivered, insights into the outcomes of care, indications of the effects of policy decisions, and data on rare diseases and operations.
Big Data of all kinds will be increasingly available for researchers. Dr. Kibbe commented that, “In the future, having a comprehensive (not sampling) country- or worldwide electronic medical record that will allow for robust inclusion of all medical data at the individual as well as cohort level will greatly contribute to the era of personalized medicine. In my opinion, this would be a real-time inclusive medical database that would allow for individual as well as population-based prospective studies.”
Dr. Haider receives support from the Henry M. Jackson Foundation of the Department of Defense, the Orthopaedic Research and Education Foundation, and the National Institutes of Health, and nonfinancial research support the Centers for Medicare and Medicaid Services Office of Minority Health. Dr. Bilimoria was the president of the Surgical Outcomes Club from 2016 to 2017.
Many are submitted, but few are chosen.
Concerned about the quality of submitted research papers based on large surgical databases that are not accepted for publication, the editorial board of JAMA Surgery has taken the initiative by giving some pointers to would-be authors. The journal editors have published a 10-point checklist of dos and don’ts to address commonly seen problems with submitted manuscripts. In addition, JAMA Surgery collaborated with the Surgical Outcomes Club to commission a series of practical guides on the most widely used data sets in an effort to improve the quality of surgical database research. The Surgical Outcomes Club is a consortium of surgeons and scientists who work to advance health services and outcomes research in surgery that was launched in 2005 at the American College of Surgeons Clinical Congress.
The authors noted that, although JAMA Surgery receives hundreds of submissions of retrospective studies of large surgical databases each year, most of these studies have flaws in the data analysis or use a hypothesis that the data sets cannot address. Hence, the editors do not send most of them out for peer review. “Of those that are sent out for peer review, many are recommended to be rejected by expert peer reviewers as they find major methodological flaws in the use of these otherwise powerful data sets,” the team wrote.
“Research using data sets can be very powerful as the research can address questions and hypotheses using large populations of people. However, the research can have many weaknesses. First, the research is only as good as the data collection for each data set. Second, the investigator needs to be familiar with the types of research questions and hypotheses that can be addressed with each data set. Third, the statistical methodology used to analyze the data is also imperative,” said Dr. Kibbe in an interview.
The checklist begins with a recommendation that the researchers develop a clear, concise hypothesis using established criteria – either FINER (for feasible, interesting, novel, ethical, relevant) or PICO (patient, population, or problem; intervention, prognostic factor, or exposure; comparison or intervention; outcome); the checklist then goes on to include compliance with institutional review board and data use agreements and to emphasize the importance of a clear take-home message that addresses policy or clinical implications.
The series comprises 11 two-page articles that aim to serve as practical guides for using each of the most widely used surgical data sets, starting with the National Inpatient Sample and ending with the Society of Thoracic Surgeons data set. Each article includes a bulleted list of the data set’s attributes, an explanation of its limitations, a history of the data set, an explanation of how the data is collected and what is unique about the set’s features, and statistical considerations researchers should take into account when analyzing the data.
The series concludes with tips from the statistical editors of JAMA Surgery – Amy H. Kaji, MD, PhD, of Harbor–University of California Los Angeles Medical Center in Torrance; Alfred W. Rademaker, PhD, of Northwestern University, Chicago; and Terry Hyslop, PhD, of Duke University, Durham, N.C. – for performing statistical analysis of large data sets: “With bigger data, random signals may denote statistical significance, and precision may be incorrectly inferred because of narrow confidence intervals,” the statistical editors noted. “While many principles apply to all studies, the importance of these methodological issues is amplified in large, complex data sets.”
However, they noted that large data sets are prone to bias and measurement errors. “It is important to respect and acknowledge the limitations of the data,” the statistical team wrote. They also reprise the introductory editorial’s call for a clear hypothesis and take-home message. “The challenge with Big Data is that it requires a carefully thought-out research question and a transparent analytic strategy,” the statistical editors said.
And there are pitfalls. Dr. Bilimoria noted, “We shouldn’t let the database define our research. We should instead be asking interesting questions and then seeking out a database that fits best to answer the question.” He said one particular problem that comes up often for reviewers is trying to discern how researchers arrived at the population of interest in a study. “A lot of inclusion and exclusions criteria are applied, and unless [the reviewer] can see the decisions that were made in the process, some fairly important biases can be introduced unintentionally. We as reviewers would like to be able to follow that exclusion pathway.”
He said, “A problem we frequently see is that these databases change the variable definitions over time – in fact, change the variables over time. So if researchers aren’t checking to see if that variable was reported the same every year of the study and in the same way, they will get spurious results. Similarly, the number of hospitals reporting is important as well since hospitals come in and out of these data sets.”
In their introductory editorial, the JAMA Surgery team noted that the checklist, practical guides, and statistical tips are a three-pronged approach that authors should consult before submitting their manuscripts. “We hope that by following these simple guides, authors can benefit from the collective wisdom of so many colleagues who have successfully completed similar analyses in the past,” they wrote.
Dr. Bilimoria sees great strengths in database research, such as giving researchers a population-level view of how care is being delivered, insights into the outcomes of care, indications of the effects of policy decisions, and data on rare diseases and operations.
Big Data of all kinds will be increasingly available for researchers. Dr. Kibbe commented that, “In the future, having a comprehensive (not sampling) country- or worldwide electronic medical record that will allow for robust inclusion of all medical data at the individual as well as cohort level will greatly contribute to the era of personalized medicine. In my opinion, this would be a real-time inclusive medical database that would allow for individual as well as population-based prospective studies.”
Dr. Haider receives support from the Henry M. Jackson Foundation of the Department of Defense, the Orthopaedic Research and Education Foundation, and the National Institutes of Health, and nonfinancial research support the Centers for Medicare and Medicaid Services Office of Minority Health. Dr. Bilimoria was the president of the Surgical Outcomes Club from 2016 to 2017.
Many are submitted, but few are chosen.
Concerned about the quality of submitted research papers based on large surgical databases that are not accepted for publication, the editorial board of JAMA Surgery has taken the initiative by giving some pointers to would-be authors. The journal editors have published a 10-point checklist of dos and don’ts to address commonly seen problems with submitted manuscripts. In addition, JAMA Surgery collaborated with the Surgical Outcomes Club to commission a series of practical guides on the most widely used data sets in an effort to improve the quality of surgical database research. The Surgical Outcomes Club is a consortium of surgeons and scientists who work to advance health services and outcomes research in surgery that was launched in 2005 at the American College of Surgeons Clinical Congress.
The authors noted that, although JAMA Surgery receives hundreds of submissions of retrospective studies of large surgical databases each year, most of these studies have flaws in the data analysis or use a hypothesis that the data sets cannot address. Hence, the editors do not send most of them out for peer review. “Of those that are sent out for peer review, many are recommended to be rejected by expert peer reviewers as they find major methodological flaws in the use of these otherwise powerful data sets,” the team wrote.
“Research using data sets can be very powerful as the research can address questions and hypotheses using large populations of people. However, the research can have many weaknesses. First, the research is only as good as the data collection for each data set. Second, the investigator needs to be familiar with the types of research questions and hypotheses that can be addressed with each data set. Third, the statistical methodology used to analyze the data is also imperative,” said Dr. Kibbe in an interview.
The checklist begins with a recommendation that the researchers develop a clear, concise hypothesis using established criteria – either FINER (for feasible, interesting, novel, ethical, relevant) or PICO (patient, population, or problem; intervention, prognostic factor, or exposure; comparison or intervention; outcome); the checklist then goes on to include compliance with institutional review board and data use agreements and to emphasize the importance of a clear take-home message that addresses policy or clinical implications.
The series comprises 11 two-page articles that aim to serve as practical guides for using each of the most widely used surgical data sets, starting with the National Inpatient Sample and ending with the Society of Thoracic Surgeons data set. Each article includes a bulleted list of the data set’s attributes, an explanation of its limitations, a history of the data set, an explanation of how the data is collected and what is unique about the set’s features, and statistical considerations researchers should take into account when analyzing the data.
The series concludes with tips from the statistical editors of JAMA Surgery – Amy H. Kaji, MD, PhD, of Harbor–University of California Los Angeles Medical Center in Torrance; Alfred W. Rademaker, PhD, of Northwestern University, Chicago; and Terry Hyslop, PhD, of Duke University, Durham, N.C. – for performing statistical analysis of large data sets: “With bigger data, random signals may denote statistical significance, and precision may be incorrectly inferred because of narrow confidence intervals,” the statistical editors noted. “While many principles apply to all studies, the importance of these methodological issues is amplified in large, complex data sets.”
However, they noted that large data sets are prone to bias and measurement errors. “It is important to respect and acknowledge the limitations of the data,” the statistical team wrote. They also reprise the introductory editorial’s call for a clear hypothesis and take-home message. “The challenge with Big Data is that it requires a carefully thought-out research question and a transparent analytic strategy,” the statistical editors said.
And there are pitfalls. Dr. Bilimoria noted, “We shouldn’t let the database define our research. We should instead be asking interesting questions and then seeking out a database that fits best to answer the question.” He said one particular problem that comes up often for reviewers is trying to discern how researchers arrived at the population of interest in a study. “A lot of inclusion and exclusions criteria are applied, and unless [the reviewer] can see the decisions that were made in the process, some fairly important biases can be introduced unintentionally. We as reviewers would like to be able to follow that exclusion pathway.”
He said, “A problem we frequently see is that these databases change the variable definitions over time – in fact, change the variables over time. So if researchers aren’t checking to see if that variable was reported the same every year of the study and in the same way, they will get spurious results. Similarly, the number of hospitals reporting is important as well since hospitals come in and out of these data sets.”
In their introductory editorial, the JAMA Surgery team noted that the checklist, practical guides, and statistical tips are a three-pronged approach that authors should consult before submitting their manuscripts. “We hope that by following these simple guides, authors can benefit from the collective wisdom of so many colleagues who have successfully completed similar analyses in the past,” they wrote.
Dr. Bilimoria sees great strengths in database research, such as giving researchers a population-level view of how care is being delivered, insights into the outcomes of care, indications of the effects of policy decisions, and data on rare diseases and operations.
Big Data of all kinds will be increasingly available for researchers. Dr. Kibbe commented that, “In the future, having a comprehensive (not sampling) country- or worldwide electronic medical record that will allow for robust inclusion of all medical data at the individual as well as cohort level will greatly contribute to the era of personalized medicine. In my opinion, this would be a real-time inclusive medical database that would allow for individual as well as population-based prospective studies.”
Dr. Haider receives support from the Henry M. Jackson Foundation of the Department of Defense, the Orthopaedic Research and Education Foundation, and the National Institutes of Health, and nonfinancial research support the Centers for Medicare and Medicaid Services Office of Minority Health. Dr. Bilimoria was the president of the Surgical Outcomes Club from 2016 to 2017.
