Problems with charting the Raptors defensive play, as a fan

When yours truly reads an article like this one …

Charting Toronto’s Defensive Play

it becomes easy to see just how problematic it is for the fanbase of this team to become sophisticated consumers of the basketball product which is put out by MLSE.

While this corner has considerable respect for an upstanding individual like Tyler Sherkin … especially, in comparison with some of the other regular “contributors” at a site that functions like … it does not mean that his observations in this article are correct and in any way present an accurate assessment of the actual defensive ability of the Raptors players, or the team in general, based upon the conclusions he’s drawn from the “stats” which he’s developed while working on this project. 

Here are some of the Main Problems which exist in this specific article:

#1. Data Invalidity – Type I: Evaluating the wrong event categories for Defense in the NBA

“Returning to the defensive stats, I was looking at a series of specific events. In particular, I tracked isolation defensive coverage, pick-and-rolls, help rotations/recoveries, contested defensive rebounds and transition defense. In the case of all but the latter, I tracked a positive and negative category. With transition defense, since the opportunities for positive events were so prolific, I only tracked negative events and there were very few, mostly because a player got fouled with no call and fell down (or flopped and fell down) and the Raps were short-handed as a result.”

Rebounding is a distinct phase within basketball and needs to be tracked as such. 

Other important defensive categories are not tracked at all, e.g. Off The Ball Positioning.

#2. Data Invalidity – Type II: Recording events in the correct categories

“The biggest problem, I found, was choosing whether to record pick-and-roll coverages by the help defender in the pick-and-roll category or the help rotation category. I unconsciously recorded several such events in different columns, so that is definitely something I have to fix the next time around by making a clear choice one way or the other.”

If the person recording these events does not have the proper Basketball Acumen and knowldege of a team’s actual strategies & tactics in use at the time, the data produced is not going to be accurate.

#3. Data Invalidity – Type III: Distinguishing accurately between a mis-applied strategy or tactic and an individual failure/breakdown

“The second biggest problem was determining how much was coaching strategy and how much was the players themselves.” 

If the person recording these events does not have the proper Basketball Acumen and knowldege of a team’s actual strategies & tactics in use at the time, the data produced is not going to be accurate.


Here are some of the Unanswered Questions which you should ask while/when/after reading this article:

I. Where’s the actual data Mr. Sherkin?

II. When using descriptive and comparative terms like “poor”, “decent” and “good”,

i. How are these categories of performance distinguished from each other?

ii. How do the Raptors players measure in these categories relative to [A] one another and [B] the other players in the NBA who play those same positions?


Here’s a list of specific conclusions which Mr. Sherkin has developed about certain Raptors, and the team in general, based on his observations from their recent games against the Pacers, the Thunder, and specifically the Bulls:

ONE - The basic assumption that Calderon is a poor individual defender was borne out.

TWO - The Raptors have but two players who are decent perimeter defenders against guys under 6’5 and those are Parker and Roko.

THREE - It is presumptuous to suggest that Anthony Parker’s individual defense is in declining. 

FOUR - The Raptors as a team did not look bad on many defensive possessions.

… while failing to account for the Main Problems [i.e. #1, #2, and #3] and the Unanswered Questions [i.e. 1 & II, i & ii, A & B] which are outlined above.

Caveat Emptor

Tags: , ,

9 Responses to “Problems with charting the Raptors defensive play, as a fan”

  1. The AltRaps Blog » Blog Archive » Toronto Raptors Linkage for April 1st through April 2nd Says:

    [...] khandor’s sports blog » Problems with charting the Raptors defensive play, as a fan – [...]

  2. Scott G Says:

    Nice critical piece. I think that most of the more general criticisms above could be leveled at most statistical analysis, particularly statistical analysis of something that is inherently extremely subjective (like Defense).

    In reality, I’m thinking that “objective” statistical analysis of defensive performances might be impossible — each can have their own opinion of the adequacy of any given defensive performance on any given possession, so I feel this makes any compilation of this information inherently suspect. As you said, caveat emptor…

  3. Linkage for Apr 1 12pm to Apr 3 5pm - Raptors Republic Says:

    [...] khandor’s sports blog » Problems with charting the Raptors defensive play, as a fan [...]

  4. Tyler Sherkin Says:

    Hah, I didn’t even see this back in April when it was published!

    I must confess, this is a very interesting read. I thought that I had effectively conveyed in my article that everything I was talking about was rough, “off the cuff” sort of stuff. A first step into the analysis, so to speak. Pretty much everything khandor says in his article about the flaws in my methods is accurate, but it wasn’t the intent at the beginning to have everything be polished and perfect right from the start. Ultimately, it was to be the beginning of a more rigorous study, with the particulars firming up as I went along.

    I didn’t end up posting the data because my recording methods were too subjective. I discovered that there was a lot more that needed to be done in order to effectively discuss the events on each possession. For example, while it’s clear that the Raptors were focused on switching every PnR like retards, that was their PLAN, so I had to reconsider if Jose getting wiped out on every pick was him sucking only, or if it was part of Triano’s retarded defensive scheme. And so on.

    As far as my descriptors, they were vague. Period. I didn’t have conclusive data, so I had subjective opinion. I saw us getting beat on the perimeter repeatedly, I saw late rotations, sometimes I saw good, hard shows from the helping big… there really isn’t a firm way to describe all that stuff unless you have a rigorous observation system in place. Something like what Riley does for the Heat, for example, which I hadn’t developed at the time. That’s one of the things coming next.

    Next up, discerning the categories. Definitely difficult. I don’t think I can effectively decide on these categories in the future without some level of decision-making before I even go into the process. As I said, I basically winged this while I was watching a series of games. Data integrity, as a result, is flawed because there were subtle differences in how I was observing each game. One of the things I should have done is recorded events, instead of subjective results. Calderon caught on the pick, PG gets into paint. That sort of thing, instead of saying labeling it poor defense. Then I could evaluate trends across players, etc.

    As to 4, that’s another aspect I didn’t get to. Focusing on the Raptors, I wanted to see what I could find about them because I don’t have access to Synergy or assistant coach-gathered scouting data or whatever, so there’s no legitimate way for me to compare to the rest of the league without a broader study. What I could do, of course, is compare the players to the guys on the opposition in the same game and see what kind of frequency with which they replicate the results the Raptors evidence. And then do that for some realy bad teams, some realy good ones and some average ones (defensively speaking), which is what I intend to do going forward.

    Then we get to the data invalidities.

    As I mentioned within my own article, there were discrete issues with the categories and with determining strategy. There’s something to be said for figuring out basketball strategy; as I said, it was clear the Raptors were intending to switch most of the time on the pick-and-roll, and that’s a strategy that had to be accounted for.

    As far as data invalidity #1 and #2, a lot of that is flat-out lack of numbers. There’s no way one person can effectively cover the entire scope of a team’s defense just watching it, not without having holes in his analysis. That’s why I chose to try and limit my observations to a certain area, and if I’m alone in my study as I go forward, I intend to continue doing that, since there’s no other way to legitimately study a team’s defense. A single person must focus on a small number of events in order to view them accurately.

    Anyway, I invite khandor to assist me in developing the study as I go forward, as it is clear that the initial approach was indeed flawed and requires much refinement. This piece of his is an excellent review of the issues I myself noticed in my own methods and I would welcome more regular input for a future attempt.

  5. khandor Says:

    Tyler Sherkin,

    1. Welcome aboard! :-)

    2. As I mentioned above, the level of respect I have for you is immense, based on:

    i. The way in which you’ve chosen to conduct yourself on-line, especially, when interacting with me;


    ii. What I’ve read from you, thus far, in terms of your work on-line.

    If I’m able to assist your efforts further … in any way, shape, or form … it would be a pleasure, from my perspective.

    I sincerely appreciate dealing with first-class people whenever this opportunity presents itself.

  6. Tyler Sherkin Says:

    Hey, you saw this! Awesome! I thought you might miss it because of the last post date on the main page.

    As a person who puts his work out for public consumption, I’ve received criticism before in many forms. The most common, invective, I ignore. When it is rational, coherent and insightful, as was yours, I can only respond positively. I happen to agree with most of your points, too, which helps.

    I’d love to go over the study and refine it with you; I enjoy this stuff a lot and you strike me as someone who would be fun and rewarding with whom to work. Any time I find people who know and love the sport, I try to get into things with them.

    Do you see the gmail address from this post? That goes straight to my Blackberry, so it’s the best way to get me. Or will be my MSN contact. I look forward to hearing from you. :)

  7. Tyler Says:

    Sent a message yesterday but I guess it got hung up in moderation or you’ve been busy, so here are the highlights:

    Your critique was polite and insightful, which is much easier to respond than the usual sort of criticism one finds putting work out there into the public’s eye. It was very easy for me to say positive things because it was so refreshing to see something of this nature, and real constructive criticism is very valuable (and much welcomed).

    You’ve got my e-mail address, and I’d love to hear what you have to say about how you might choose to go about rebuilding the study, even if it turns into something a little more limited at first. A series of studies on, for example, wing pick-and-rolls, point guard defense, post defense, etc, etc, breaking each study into a more isolated category with fewer variables to track. Or whatever else you come up with. :)

  8. khandor Says:


    Sorry for the initial delay.

    Check your email account. :-)


  9. Tyler Says:

    No worries, I thought that something errored out or something, so I sent it again. At first, I didn’t actually realize that the comments section was moderated, so…

Leave a Reply