[Maths-Education] Re: ICT in mathematics

[Alan Rogerson]

Dear David,

May I (again) try to clarify that there has been absolutely no negative 
criticism of the merit of the considerable body of work before and after 
the establishment of Carnegie.inc? There is no need therefore "to be 
fair" since what you, very helpfully, describe is all about the long and 
impressive history of the original work at Carnegie Mellon University 
(not Melon, but hey it's not an "l" of a difference).

I agree very much, however, that "it would be interesting to know if 
"independent" evaluations of the effectiveness have been undertaken and 
published. " That was precisely my point, I don't know of any such work, 
and it would help us to evaluate this work of the creators, originators 
and eventually sellers of this material. It was very helpful to have 
your own impressions and evaluations. It was good to see too that your 
link below was to "The National Center for Academic Transformation 
(NCAT) is an independent, not-for-profit organization" so their 
evaluation of the work at Northern Arizona University looks truly 
independent except..oops,... it appears to be a direct quote from the 
NAU itself, about their own courses! The discussion of bias covers 
everything, not just commercial companies, in reality most if not all 
institutions would be positively biassed to their own way of doing 
things - so is there any chance of an independent evaluation here too??

Thanks for your personal evaluation of the Cognitive Tutor, which we can 
take to be independent and from experience - very helpful to Sarah who 
we hope is following this thread and taking notes (at least from you!). 
Over the years I have looked at a myriad of ICT programs, and would have 
to assess the majority as poor to very poor. It is therefore all the mor 
exciting and important when we find, and disseminate news about, those 
programs that are innovative and (from a subjective view of course) 
appear to work well and genuinely enhance the learning of our students. 
I am at this very moment evaluating at the request of the creators a 
very interesting and complex program called.. wait for it Statistics for 
the Terrified, which is still being trialled. If anyone on the list is 
interested in that area (and is we presume sufficiently terrified) 
please contact Jill at smorris at conceptstew.co.uk. This is I would say 
good (but not perfect, hence the trialling) ICT.

Best wishes, and sorry if it's Friday,
Alan



On 04/03/2011 17:57, David H Kirshner wrote:
> ***********************************************************************************************************
> This message has been generated through the Mathematics Education email discussion list.
> Hitting the REPLY key sends a message to all list members.
> ***********************************************************************************************************
> I'm enjoying this thread.
>
> To be fair, Ken Koedinger, lead developer of the software, is a bona
> fide cognitive scientist who has published a variety of academic papers
> on the cognitive theory involved, in addition to publications on the
> effectiveness of the tutor. And even the latter publications have been
> in peer review academic journals. Still, it would be interesting to know
> if "independent" evaluations of the effectiveness have been undertaken
> and published.
>
> There is some software on the market now that seems to be effective in
> raising students' test scores in comparison with standard classroom
> instruction. For instance, at Northern Arizona University, the college
> algebra course is taught through an online facility that
>
> "promote[s] active and collaborative learning, address the diverse range
> of student learning styles and permit acceleration. It will use ALEKS, a
> comprehensive instructional software and course management system that
> individualizes assessment and learning. ALEKS is Web-based, allowing
> students to access course materials at anytime from anyplace, and can be
> customized to meet student and course needs. As a supplement to ALEKS,
> NAU will also use Thinkwell, a video-based mathematics software program
> that uses highlighted worked examples and video lectures for students
> who learn better in this mode."
> (http://www.thencat.org/PCR/R3/NAU/NAU_Abstract.htm)
>
> However, it is clear that the program's effectiveness primarily is
> related to its effective provision of practice problems and immediate
> feedback. In short, it promotes learning of skills through repetitive
> practice in the behaviorist tradition.
>
> I've looked at the Cognitive Tutor, and it's a more complex and
> ambitious project. The Tutor functions through a multidimensional model
> of expertise that it uses to update a student model, and thereby
> determine what kinds of problems and assistance would best benefit the
> student. It is not so straightforward an analysis as to whether what is
> being promoted is skills through practice (in a sophisticated way), or
> something more akin to what we would consider as mathematical concepts.
>
> The Cognitive Tutor is based on the ACT-series of acquisition models
> developed by John Anderson and company at Carnegie Melon--in fact, Ken
> was a student of his, and is now a frequent collaborator. The ambiguity
> of the Tutor in terms of its learning modalities is reflected also in
> the status of the ACT theories. Anderson (2005) would be a good source
> to begin to answer those questions, and Anderson (2007) includes an
> interesting discussion of the general issue of the kind of learning
> addressed in the ACT theories.
>
> David Kirshner
>
> Anderson, J. R. (2005). Human symbol manipulation within an integrated
> cognitive architecture. Cognitive Science, 29, 313-341. [algebraic
> symbol skills]
> Anderson, J. R. (2007). How can the human mind occur in the physical
> universe?. Oxford University Press.
>
>
>
>
> -----Original Message-----
> From: maths-education-bounces at lists.nottingham.ac.uk
> [mailto:maths-education-bounces at lists.nottingham.ac.uk] On Behalf Of
> Alan Rogerson
> Sent: Friday, March 04, 2011 10:10 AM
> To: Mathematics Education discussion forum
> Subject: [Maths-Education] Re: ICT in mathematics
>
> ************************************************************************
> ***********************************
> This message has been generated through the Mathematics Education email
> discussion list.
> Hitting the REPLY key sends a message to all list members.
> ************************************************************************
> ***********************************
> Dear Dylan,
>
> What you say below does not in any way alter the fact that what you
> actually recommended to Sarah were reports on a webpage produced by a
> commercial company. That is the problem. You say something is "one of
> the best researched" but we need to know when and by whom? It is not the
>
> quantity of research that counts, rather its quality.
>
> Please note, I am not making any judgement about the actual research
> which you call "original" nor did I say that this research is some how
> invalidated by being used by a commercial company. Let's say for the
> sake of argument, that  all this research could be validated, and also
> note that some of the reports on the Carnegie.inc webpage were (as we
> know) from Carnegie Mellon University itself, and may have even
> pre-dated the formation of Carnegie.inc, I do hope you can see that this
>
> does not change the problem? "Selective quotation" is still a real
> hazard, what company after all will quote research critical of its own
> products?
>
> We know only too well the much bigger and much more serious debate going
>
> on about so-called academic research being funded, or supported, (or of
> course suppressed!) by drug companies. Companies are in business to make
>
> money, so we can hardly use them, or the reports they quote, as
> objective exemplars of "research". The contrast is between reports which
>
> clearly have no such bias, and those which are at risk of being biassed.
>
> Surely we cannot say "third party evaluations... would be better",
> surely you mean essential? We know from basic statistics that biassed
> evidence, when we can  not attach boundaries to the bias, , is, and must
>
> be, useless (not second best). We all know the story of the millions of
> telephone calls surveyed  that failed to predict the next President of
> the USA....?
>
> Please also note that there is absolutely no bias (or specific
> accusations) against Carnegie.inc in particular here, it is a purely
> general point that is being made.
>
> The only remaining problem, and somewhat insoluble, is the one Douglas
> Butler has just mentioned.
>
> C'est la vie, c'est la ICT.
>
> Best wishes,
> Alan
>
>
>
>
> On 04/03/2011 15:52, dylanwiliam at mac.com wrote:
>> Alan: Sarah asked specifically for studies that showed the impact of
> ICT on attainment, and the Cognitive Tutor is one of the best researched
> pieces of software for mathematics education.  While Carnegie Learning
> is a commercial company that has taken over marketing and distribution
> of the products generated by the people who developed the Cognitive
> Tutor, the research itself is very solid (and much of it dates from
> before Carnegie Learning became involved). I agree that third party
> evaluations, such as those undertaken by Mathematica, would be better,
> and of course educationalists should evaluate the merits of the studies,
> but the fact that the research is now being used to support a commercial
> enterprise does not invalidate the original findings.
>> Dylan
>
> This message and any attachment are intended solely for the addressee
> and may contain confidential information. If you have received this
> message in error, please send it back to me, and immediately delete it.
> Please do not use, copy or disclose the information contained in this
> message or in any attachment.  Any views or opinions expressed by the
> author of this email do not necessarily reflect the views of the
> University of Nottingham.
>
> This message has been checked for viruses but the contents of an
> attachment
> may still contain software viruses which could damage your computer
> system:
> you are advised to perform your own checks. Email communications with
> the
> University of Nottingham may be monitored as permitted by UK
> legislation.
>
>
>