rbnn

09-01-05, 10:54 AM

Are there any studies analyzing, within a season, the effect of individual umpires on pitchers' ERA?

View Full Version : Umpire's effect on ERA?

rbnn

09-01-05, 10:54 AM

Are there any studies analyzing, within a season, the effect of individual umpires on pitchers' ERA?

pedromartinezfan

09-01-05, 11:04 AM

Maybe this helps: http://www.baseballprospectus.com/statistics/umpire_report2005.php

JDPNYY

09-01-05, 11:18 AM

I don't wanna be pitching the next time Dusty Dellinger works Home Plate.

rbnn

09-01-05, 11:29 AM

Maybe this helps: http://www.baseballprospectus.com/statistics/umpire_report2005.php

That is helpful and interesting, thank you.

However, that chart only lists the average ERA for an umpire. What we are after is the effect of an umpire on ERA. The umpire might be seeing pitchers of different ERAs.

The simplest way to do this is to ask for the average ERA of all the pitchers an umpire has seen, and compare that to his ERA on that chart. But there are more precise ways to do this.

Still, we do see that Ron Kulpa, the ump in the RJ shutout last night, had lower ERA of 3.96 than Dale Scott, the ump in the Mussina game, who's ERA was 4.41.

That is helpful and interesting, thank you.

However, that chart only lists the average ERA for an umpire. What we are after is the effect of an umpire on ERA. The umpire might be seeing pitchers of different ERAs.

The simplest way to do this is to ask for the average ERA of all the pitchers an umpire has seen, and compare that to his ERA on that chart. But there are more precise ways to do this.

Still, we do see that Ron Kulpa, the ump in the RJ shutout last night, had lower ERA of 3.96 than Dale Scott, the ump in the Mussina game, who's ERA was 4.41.

WHIP

09-01-05, 11:33 AM

That is helpful and interesting, thank you.

However, that chart only lists the average ERA for an umpire. What we are after is the effect of an umpire on ERA. The umpire might be seeing pitchers of different ERAs.

The simplest way to do this is to ask for the average ERA of all the pitchers an umpire has seen, and compare that to his ERA on that chart. But there are more precise ways to do this.

Still, we do see that Ron Kulpa, the ump in the RJ shutout last night, had lower ERA of 3.96 than Dale Scott, the ump in the Mussina game, who's ERA was 4.41.

As I said in another thread there are far too many confounding variables for such an analysis to be meaningful. As for your final point, Bruce Froemming is known as a tight, regulation-zone umpire and he gave RJ lots of grief earlier in the year against Boston. And according to this chart his ERA is 3.85, lower than Kulpa's. When I gamble I do take the umpire into consideration if the pitcher's notorious for control issues. However the data's too random and the confounding variables are too enormous for an actual analysis.

However, that chart only lists the average ERA for an umpire. What we are after is the effect of an umpire on ERA. The umpire might be seeing pitchers of different ERAs.

The simplest way to do this is to ask for the average ERA of all the pitchers an umpire has seen, and compare that to his ERA on that chart. But there are more precise ways to do this.

Still, we do see that Ron Kulpa, the ump in the RJ shutout last night, had lower ERA of 3.96 than Dale Scott, the ump in the Mussina game, who's ERA was 4.41.

As I said in another thread there are far too many confounding variables for such an analysis to be meaningful. As for your final point, Bruce Froemming is known as a tight, regulation-zone umpire and he gave RJ lots of grief earlier in the year against Boston. And according to this chart his ERA is 3.85, lower than Kulpa's. When I gamble I do take the umpire into consideration if the pitcher's notorious for control issues. However the data's too random and the confounding variables are too enormous for an actual analysis.

rbnn

09-01-05, 11:53 AM

As I said in another thread there are far too many confounding variables for such an analysis to be meaningful. As for your final point, Bruce Froemming is known as a tight, regulation-zone umpire and he gave RJ lots of grief earlier in the year against Boston. And according to this chart his ERA is 3.85, lower than Kulpa's. When I gamble I do take the umpire into consideration if the pitcher's notorious for control issues. However the data's too random and the confounding variables are too enormous for an actual analysis.

The presence of multiple extraneous confounding variables could be accounted for in the significance level of any conclusions drawn, as is standard in any regression.

The presence of multiple extraneous confounding variables could be accounted for in the significance level of any conclusions drawn, as is standard in any regression.

WHIP

09-01-05, 12:09 PM

The presence of multiple extraneous confounding variables could be accounted for in the significance level of any conclusions drawn, as is standard in any regression.

Account for them here, then. Let's review of some of the confounding variables:

1) Different pitchers

2) Pitchers pitch well one day and do not pitch well other days

3) Different lineups

4) Slumping and hot lineups

5) Home/road splits

6) Problems with ERA as a predictively useful metric

7) Randomness - to use a current example, the data we've been investigating, Kulpa's ERA last year was 4.64 as compared to 2005's 3.96. Froemming, whom I mentioned earlier, is 5.43 to 3.85. Tremendous fluctuations there.

These variables are anything but extraneous, and there are way too many of them to assign statistical significance to an umpire's influence.

Account for them here, then. Let's review of some of the confounding variables:

1) Different pitchers

2) Pitchers pitch well one day and do not pitch well other days

3) Different lineups

4) Slumping and hot lineups

5) Home/road splits

6) Problems with ERA as a predictively useful metric

7) Randomness - to use a current example, the data we've been investigating, Kulpa's ERA last year was 4.64 as compared to 2005's 3.96. Froemming, whom I mentioned earlier, is 5.43 to 3.85. Tremendous fluctuations there.

These variables are anything but extraneous, and there are way too many of them to assign statistical significance to an umpire's influence.

rbnn

09-01-05, 12:52 PM

I fail to understand your terminology or analysis, but will try to simplify things for you nonetheless in order that you will understand the matter better.

Where multiple variables can affect a particular outcome, numerous statistical methodologies exist to isolate the effect of one variable on that outcome. Such methodologies include regression and principal component (PCA) analysis.

In assessing the effect of a particular variable, such an analysis will include an estimate of the statistical significance of the result. Numerous tests for statistical signfiicance exist. Hidden variables not accounted for in the model can cause anomalies in the analysis, which is one reason data analysis requires domain specific knowledge in many cases. But a hidden variable - such as weather, for example - is likely not a problem here for one simple reason: DATA.

Data analysis is also handicapped because it fundamentally is concerned with correlative effects, whereas we are interested in causative effects: does an umpire affect pitchers' ERA and if so how much? (Data analysis would answer the question: is there a correlation between umpire and ERA; in causation analysis hidden variables are more of a problem).

The presence of voluminous high-quality data should allow a PCA or regression analysis to perform very well. Here is how.

Construct a table showing for each pitch thrown in each game, each of the variables that you mention below: pitcher's ERA, result (hit or out), call (ball or strike), OBP of batter, umpire, and so on.

Your first task is simply to determine whether the umpire does affect the result or the call. This is just a multiple regression of the input variables - ERA, OBP, etc. - versus the output variables (result of AB and call on pitch). Many standard stats packages can answer this question and assess the signficance level they ascribe. For example Matlab is my favorite, although many hardcore stats people use S.

With many variables, you can use principal components analysis to select a few of the most important input variables in the prediction.

The key here is that if in fact umpiring is a poor predictor of hit result given the other data, because of the huge mass of data you have it will show up easily enough.

Now I will address specifically your variables.

Account for them here, then. Let's review of some of the confounding variables:

1) Different pitchers

This is not a confounding variable, this is an input variable accounted for in the model, and it is exactly why I said above, the aggregate ERA reported by baseballprospectus is not that useful.

2) Pitchers pitch well one day and do not pitch well other days

True, but the distribution of such days should be independent of the distribution of HP umpires. Thus, this is not a confounding variable, it is an extraneous variable, and in any case would not affect the results. Conversely, if pitchers pitching well were NOT independent of umpire but was CORRELATED with umpire, then there is likely a causal link that we are trying to elicit.

3) Different lineups

Again, if the lineup is INDEPENDENT of the umpire, then it will not affect the conclusion. If the lineup is CORRELATED to the umpire, which seems improbable, then there is indeed a hidden variable problem.

4) Slumping and hot lineups

See 3.

5) Home/road splits

This is accounted for by adding an input variable of "home/road" in the regression. For that matter, even stadium could be added.

6) Problems with ERA as a predictively useful metric

This is true, but again, errors in ERA should not be correlated particularly with umpire identity.

7) Randomness - to use a current example, the data we've been investigating, Kulpa's ERA last year was 4.64 as compared to 2005's 3.96. Froemming, whom I mentioned earlier, is 5.43 to 3.85. Tremendous fluctuations there.

First, I said in the post beginning the thread that I only wanted correlations within a season, not between seasons. So that resolves this entire problem.

These variables are anything but extraneous, and there are way too many of them to assign statistical significance to an umpire's influence

Finally, I emphasize that it may well be such analysis concludes umpire is not causally related to ERA or pitcher performance. I do not claim there is a relation. I only claim that in view of what appears to be both widely differing strike zones among umpires and a strong effect of such strike zones on some pitchers, it is reasonable to investigate any correlative or causal link between umpire and pitcher ERA. Such investigation requires for each pitch the data above (although one could get OK results just for each AB the data above).

I would have expected analysis more pinpoint than the very rough chart on baseballprospectus would have been done, as it's easy and natural to do it.

Where multiple variables can affect a particular outcome, numerous statistical methodologies exist to isolate the effect of one variable on that outcome. Such methodologies include regression and principal component (PCA) analysis.

In assessing the effect of a particular variable, such an analysis will include an estimate of the statistical significance of the result. Numerous tests for statistical signfiicance exist. Hidden variables not accounted for in the model can cause anomalies in the analysis, which is one reason data analysis requires domain specific knowledge in many cases. But a hidden variable - such as weather, for example - is likely not a problem here for one simple reason: DATA.

Data analysis is also handicapped because it fundamentally is concerned with correlative effects, whereas we are interested in causative effects: does an umpire affect pitchers' ERA and if so how much? (Data analysis would answer the question: is there a correlation between umpire and ERA; in causation analysis hidden variables are more of a problem).

The presence of voluminous high-quality data should allow a PCA or regression analysis to perform very well. Here is how.

Construct a table showing for each pitch thrown in each game, each of the variables that you mention below: pitcher's ERA, result (hit or out), call (ball or strike), OBP of batter, umpire, and so on.

Your first task is simply to determine whether the umpire does affect the result or the call. This is just a multiple regression of the input variables - ERA, OBP, etc. - versus the output variables (result of AB and call on pitch). Many standard stats packages can answer this question and assess the signficance level they ascribe. For example Matlab is my favorite, although many hardcore stats people use S.

With many variables, you can use principal components analysis to select a few of the most important input variables in the prediction.

The key here is that if in fact umpiring is a poor predictor of hit result given the other data, because of the huge mass of data you have it will show up easily enough.

Now I will address specifically your variables.

Account for them here, then. Let's review of some of the confounding variables:

1) Different pitchers

This is not a confounding variable, this is an input variable accounted for in the model, and it is exactly why I said above, the aggregate ERA reported by baseballprospectus is not that useful.

2) Pitchers pitch well one day and do not pitch well other days

True, but the distribution of such days should be independent of the distribution of HP umpires. Thus, this is not a confounding variable, it is an extraneous variable, and in any case would not affect the results. Conversely, if pitchers pitching well were NOT independent of umpire but was CORRELATED with umpire, then there is likely a causal link that we are trying to elicit.

3) Different lineups

Again, if the lineup is INDEPENDENT of the umpire, then it will not affect the conclusion. If the lineup is CORRELATED to the umpire, which seems improbable, then there is indeed a hidden variable problem.

4) Slumping and hot lineups

See 3.

5) Home/road splits

This is accounted for by adding an input variable of "home/road" in the regression. For that matter, even stadium could be added.

6) Problems with ERA as a predictively useful metric

This is true, but again, errors in ERA should not be correlated particularly with umpire identity.

7) Randomness - to use a current example, the data we've been investigating, Kulpa's ERA last year was 4.64 as compared to 2005's 3.96. Froemming, whom I mentioned earlier, is 5.43 to 3.85. Tremendous fluctuations there.

First, I said in the post beginning the thread that I only wanted correlations within a season, not between seasons. So that resolves this entire problem.

These variables are anything but extraneous, and there are way too many of them to assign statistical significance to an umpire's influence

Finally, I emphasize that it may well be such analysis concludes umpire is not causally related to ERA or pitcher performance. I do not claim there is a relation. I only claim that in view of what appears to be both widely differing strike zones among umpires and a strong effect of such strike zones on some pitchers, it is reasonable to investigate any correlative or causal link between umpire and pitcher ERA. Such investigation requires for each pitch the data above (although one could get OK results just for each AB the data above).

I would have expected analysis more pinpoint than the very rough chart on baseballprospectus would have been done, as it's easy and natural to do it.

WHIP

09-01-05, 04:18 PM

Ok, thanks for explaining - if you ever actually perform a study, be sure to let us know.

rbnn

09-01-05, 04:26 PM

Ok, thanks for explaining - if you ever actually perform a study, be sure to let us know.

As I say, it's surely been done.

What I don't have is the raw data in reasonably manipulable form. Is there a spreadsheet somewhere that shows the result of each at-bat for each game, including who was pitching, batting, result of each pitch, and so on? Obviously Elias has it, but is there a free source?

I'm curious, realistically I don't have time to do this though.

As I say, it's surely been done.

What I don't have is the raw data in reasonably manipulable form. Is there a spreadsheet somewhere that shows the result of each at-bat for each game, including who was pitching, batting, result of each pitch, and so on? Obviously Elias has it, but is there a free source?

I'm curious, realistically I don't have time to do this though.

Powered by vBulletin™ Version 4.0.7 Copyright © 2019 vBulletin Solutions, Inc. All rights reserved.