tag:blogger.com,1999:blog-9668863771990453972024-03-13T12:10:54.102+00:00Magical Linux ArticlesEverything Linux and open sourceAndy Pietershttp://www.blogger.com/profile/02919532566299594342noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-966886377199045397.post-84292430638603701182018-09-12T19:47:00.000+01:002018-09-30T17:34:01.951+01:00BASH: singular or plural done easy, a story of ternaries<style>
li.L0, li.L1, li.L2, li.L3,
li.L5, li.L6, li.L7, li.L8 {
list-style-type: decimal !important;
}
pre.output { background-color: black; color: lime; border: 3px inset #ffbf00; }
p.quote { border: 1px solid black; white-space: pre; }
</style>
<h2>The introduction</h2>
<p>To avoid looking silly and being made fun the world over for having made a program that prints hair-raising butchered English like <cite>Copying 1 files</cite>, many programming languages have ternary constructs.</p>
<p>in PHP for instance one can use the ternary operator, <strong>?</strong>:</p>
<pre class="prettyprint lang-php"><?php
$intFiles=1;
printf("Copying %d file%s",$intFiles,1==$intFiles?null:'s');
?></pre>
<p>BASH is often insulted for having awkward and arcane syntax and as far as ternaries are concerned it doesn't disappoint. The closest equivalent is the case construct:</p>
<h2>The first solution</h2>
<pre class="prettyprint lang-bsh">
file=1
echo -n "Copying ${file} "
case "${file}" in
1) echo "file" ;;
*) echo "files" ;;
esac
</pre>
<p>There are some problems with this approach. Not only do you have to split up your printing into several parts you also have to have a case construct for each test. What if you wanted to construct a sentence with several items being counted? Can you spell tiresome?</p>
<p>Luckly BASH, awkward arcane that it is, has a more florid solution, in the form of parameter expansion:</p>
<p class="quote">${parameter:-word}
Use Default Values. If parameter is unset or null, the expansion of word is substituted.
Otherwise, the value of parameter is substituted.</p>
<p>In other words, using this expansion you can print the value of a variable, or, a default value if the variable isn't defined or set</p>
<h2>A better solution</h2>
<pre class="prettyprint lang-bsh">
word=([1]=file)
number=1
echo "Copying ${word} ${word[${number}]:-files}"
</pre>
<p>OK, as long as you manage to keep track of the tongue-twisting mix of curly braces and brackets it is surprisingly easy and compact, at least when compared to the case construct.</p>
<p>Let's put it in a small script and see how it behaves with some edge cases:</p>
<pre class="prettyprint lang-bsh linenums">#!/bin/bash
# save as /tmp/newcats.sh
catword=(
[1]=cat
)
while `true`; do
read -p "How many? " cats || break
echo "You have seen ${cats} ${catword[${cats}]:-cats}"
done
</pre>
<pre class="output">$ sh /tmp/newcats.sh
How many? 0
You have seen 0 cats
How many? 1
You have seen 1 cat
How many? 2
You have seen 2 cats
How many? -1
You have seen -1 cat
How many? -2
You have seen -2 cats
How many? a
You have seen a cats
How many? @
/tmp/newcats.sh: line 13: @: syntax error: operand expected (error token is "@")
How many? 1.5
/tmp/newcats.sh: line 13: 1.5: syntax error: invalid arithmetic operator (error token is ".5")
How many? a few
/tmp/newcats.sh: line 13: a few: syntax error in expression (error token is "few")</pre>
<p>Not bad, we see that negatives are handled correctly because for negative subscripts BASH substracts them from the total array count. So -1 means the highest item and since we have only one item, -1 is treated as 1, and -anything else is treated as plural. Great!</p>
<p>We do fall apart when it comes to floats and gibberish because indexed arrays need an integer subscript.</p>
<p>The obvious solution, therefore, is to use an associative array:</p>
<h2>Solution the third</h2>
<p>For associative arrays, BASH will not perform arithmetic on its subscripts so the handy trick with negatives will not work any more. This is however easily remedied by explicitly defining a -1 item.</p><p>BASH also requires explicit declaration of associative arrays through the -A flag.</p>
<pre class="prettyprint lang-bsh linenums">#!/bin/bash
# save as /tmp/coolcats.sh
declare -A catword=(
[1]=cat
[-1]=cat
)
while `true`; do
read -p "How many? " cats || break
echo "You have seen ${cats} ${catword[${cats}]:-cats}"
done
</pre>
<pre class="output">$ sh /tmp/coolcats.sh
How many? 0
You have seen 0 cats
How many? 1
You have seen 1 cat
How many? 2
You have seen 2 cats
How many? -1
You have seen -1 cat
How many? -2
You have seen -2 cats
How many? a
You have seen a cats
How many? @
You have seen @ cats
How many? 1.5
You have seen 1.5 cats
How many? a few
You have seen a few cats
How many?
/tmp/coolcats.sh: line 14: catword: bad array subscript
You have seen cats
How many? %#$%^#[]
You have seen %#$%^#[] cats
How many? '"`:{}
You have seen '"`:{} cats</pre>
<p>Excellent! We are handling all cases correctly and the only thing we choke on is an empty input. That is easily helped though by using the same technique on the variable referencing the index.
<h2>Putting it all together</h2>
<pre class="prettyprint lang-bsh">#!/bin/sh
# save as /tmp/ubercat.sh
declare -A catword=(
[-1]=cat
[1]=cat
)
declare -A is=(
[-1]=is
[1]=is
)
while `true`; do
read -p "How many? " cats || break
echo "There ${is[${cats:-0}]:-are} ${cats:-0} ${catword[${cats:-0}]:-cats}"
done</pre>
<pre class="output">~]$ sh /tmp/ubercat.sh
How many?
There are 0 cats
How many? 1
There is 1 cat
How many? 1.5
There are 1.5 cats
How many? -1
There is -1 cat
How many? -1.5
There are -1.5 cats
How many? a few
There are a few cats
How many? %#@%$^!
There are %#@%$^! cats</pre>
<h2>In conclusion</h2>
<p>BASH is a versatile scripting language and although its syntax <strong>is</strong> awkward and possibly arcane, it can quite hold its own when it comes to manipulating strings. In this age of ever faster computers with more and more memory one is often seduced to outsourcing functionality in BASH scripts to utilities such as <em>sed</em> and <em>awk</em> and even full-size interpreters like <em>perl</em> and <em>php</em>. Digging into the BASH manual is daunting. The thing is a behemoth containing more than 100 pages of dry terse language. Spend some time experimenting however and you will discover some real gems and ultimately, doing things in the language is always going to be more efficient than doing it externally.</p>Andy Pietershttp://www.blogger.com/profile/02919532566299594342noreply@blogger.com0tag:blogger.com,1999:blog-966886377199045397.post-37831838106657159072018-09-12T18:22:00.001+01:002022-07-16T17:20:45.792+01:00BASH pointer juggling<style>
li.L0, li.L1, li.L2, li.L3,
li.L5, li.L6, li.L7, li.L8 {
list-style-type: decimal !important;
}
pre.output { background-color: black; color: lime; border: 3px inset #ffbf00; }
p.quote { border: 1px solid black; white-space: pre; }
strong.yellow u { background-color: yellow; color: black; }
</style>
<h2>The introduction</h2>
<p>BASH, like many programming and scripting languages, has pointer types. The word pointer says it all, it points to something else, it <em>references</em> something. You can use the contents of any scalar value in BASH as reference by using the <code>${!varname}</code> syntax.</p>
<p>To create an actual pointer variable though you need to <code>declare -n</code> it.</p>
<p>Once you declare a variable as pointer you can use the <code>${!varname}</code> syntax to find out where it points to</p>
<h2>An example</h2>
<pre class="prettyprint lang-bsh">
#!/bin/bash
# save as /tmp/bashpointer.sh
targetVariable="the_target"
the_target="the value being pointed at"
echo "The contents of the ${targetVariable} variable is '${!targetVariable}'"
declare -n aRealPointer="the_target"
echo "The contents of the ${!aRealPointer} variable is '${aRealPointer}'"
</pre>
<pre class="output">$ sh /tmp/bashpointer.sh
The contents of the the_target variable is 'the value being pointed at'
The contents of the the_target variable is 'the value being pointed at'
</pre>
<p>A task that I come across often is the processing of configuration files. We check if a variable is defined and copy it to the actual variable used throughout the script. This job is quite tedious and error-prone for anything more than a single variable. </p>
<p>This can be solved easily enough though with BASH pointers.</p>
<p>Let's make an array where each even-numbered index contains the variable to set, and each odd-numbered index contains the variable to read:</p>
<pre class="prettyprint lang-bsh">
CHECKS=(
"name" "profile_name"
"value" "profile_value"
)
</pre>
<p>So we go through the list and check if the odd-numbered variable is set and if it is copy it to the even-numbered variable:</p>
<h2>A first attempt</h2>
<pre class="prettyprint lang-bsh linenums">
#!/bin/bash
# save as /tmp/refs.sh
profile_name="The name of the profile"
profile_value="The value of the profile"
name="___name"
value="___value"
CHECKS=(
"name" "profile_name"
"value" "profile_value"
)
set - ${CHECKS[@]}
typeset -n theref
while [ $# -gt 0 ]; do
theref=$1
[ -n "${!2}" ] && theref="$2"
shift 2
done
echo " Name: ${name}"
echo "Value: ${value}"
</pre>
<pre class="output">$ sh /tmp/refs.sh
Name: The value of the profile
Value: ___value
</pre>
<p>That is unexpected! What is going on? The <em>Value</em> variable did not get set, and the <em>Name</em> variable has got the wrong value!</p>
<p>If we analyse this a bit more though, we can make the following observations: </p>
<ul>
<li>The value of the <em>Name</em> variable changed therefore we have successfully created a pointer to the <em>Name</em> variable</li>
<li>The value meant for the <em>Value</em> variable has been put in the <em>Name</em> variable</li>
<li>The <em>Value</em> variable has not changed which means we have never been able to create a pointer to it</li>
</ul>
<p>All of these observations spell out one thing: <strong>Once set, the pointer's target has not changed</strong></p>
<p>This is corroborated by the BASH manual for <code>declare -n</code>:
<p class="quote">
declare [-aAfFgilnrtux] [-p] [name[=value] ...]
typeset [-aAfFgilnrtux] [-p] [name[=value] ...]
...
-n Give each name the nameref attribute, making it a name reference to another variable. That other variable is defined by the value of name. <strong class="yellow">All references,
assignments, and attribute modifications to name, <u>except those using or changing the -n attribute itself</u>, are performed on the variable referenced by name's value.</strong>
The nameref attribute cannot be applied to array variables.
</p>
<p>OK, according to this we can only update the value being pointed at by removing the <code>-n</code> attribute first:</p>
<h2>Second attempt</h2>
<pre class="prettyprint lang-bsh">
#!/bin/bash
# save as /tmp/refs2.sh
profile_name="The name of the profile"
profile_value="The value of the profile"
name="___name"
value="___value"
CHECKS=(
"name" "profile_name"
"value" "profile_value"
)
set - ${CHECKS[@]}
while [ $# -gt 0 ]; do
typeset -n theref="$1"
[ -n "${!2}" ] && theref="$2"
shift 2
typeset +n theref
done
echo " Name: ${name}"
echo "Value: ${value}"</pre>
<pre class="output">
$ sh /tmp/refs2.sh
Name: profile_name
Value: profile_value
</pre>
<p>Success!!</p>
<h2>In conclusion</h2>
<p>There is no understating the merrits of pointers, yet the age old addage about great power and responsibility applies here too. Being responsible with BASH pointers means flipping the <code>-n</code> property prior to changing its target, <strong>or</strong>, using a local variable that goes out of scope between calls</p>.Andy Pietershttp://www.blogger.com/profile/02919532566299594342noreply@blogger.com0tag:blogger.com,1999:blog-966886377199045397.post-19392878321573353062013-02-04T17:29:00.001+00:002013-02-04T17:30:15.019+00:00Announcing... Plasma wallpaper plugin<h2>
ExternalCommand 1.0.0</h2>
<div>
This is a plasma wallpaper that plugs into the standard Desktop Settings configuration widget and allows you to choose a command that will be periodically called to provide the name of the next wallpaper to display.</div>
<div>
<br /></div>
<div>
This enables you to get virtually any dynamic or static content as a wallpaper, even if you don't know programming, you can for instance, use the curl or wget commands to retrieve Wikipedia or Flickr's image of the day.</div>
<div>
<br />
My first C++ and Qt project, hope you enjoy it.<br />
<br /></div>
<div>
The project is hosted on Sourceforge: <a href="http://externalcommand.sourceforge.net/" target="_blank">http://externalcommand.sourceforge.net</a> </div>
<div>
<br /></div>
<div style="text-align: center;">
Screen shots:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-SmahT5bXLUU/UQ_vTWutRQI/AAAAAAAAAVU/7zNv516QY1s/s1600/Configure.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-SmahT5bXLUU/UQ_vTWutRQI/AAAAAAAAAVU/7zNv516QY1s/s1600/Configure.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-v-lhDDiPPjk/UQ_vTeJTPaI/AAAAAAAAAVY/FWleVQ81neU/s1600/SetCommand.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-v-lhDDiPPjk/UQ_vTeJTPaI/AAAAAAAAAVY/FWleVQ81neU/s1600/SetCommand.png" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
Andy Pietershttp://www.blogger.com/profile/02919532566299594342noreply@blogger.com0tag:blogger.com,1999:blog-966886377199045397.post-7086076373271881142012-07-30T21:17:00.000+01:002018-09-30T17:52:10.448+01:00On MySQL Merge tables, or how to fix the dreaded "defined differently" error.<style>
span.highlight { color: #aadd99; background-color: black; }
span.yellow-reverse { background-color: yellow; color: black; }
</style>
The MySQL Merge storage engine is nothing more than a means to link together identical MyISAM tables into one big one that you can query at your leisure.
To create one, first create a table with a structure identical to one of the tables you want to link, for instance by doing a <tt>create</tt>-like statement.
The next, and final step, is to alter the new table's engine to merge and define the union.
Simple as pie.
It won't be long, however, before MySQL throws an error at you:<br />
<br />
<pre style="background-color: black; color: lime; font-weight: bold; text-align: center">
ERROR 1168 (HY000): Unable to open underlying table which is
differently defined or of non-MyISAM type or doesn't exist</pre>
<p>
The causes outlined below should cover 98% of all the situations where you encounter this error.</p>
<p>
<ol>
<li><span class="highlight">The tables really are different</span><br />
This may be because the schema of one of the tables is different, or was created with another MySQL version (and you copied the raw tables across at one point)</li>
<li><span class="highlight">Some, or all of the tables are not MyISAM tables</span><br />
Merge only works with MyISAM tables so the solution in this case is to change the storage engine type on all the tables to MyISAM</li>
<li><span class="highlight">Some, or all of the tables in your union do not exist</span><br />MySQL does not check if the tables exist when you create the UNION, so it is perfectly possible to create a UNION on tables that don't exist, and MySQL will only tell you when you try to use the merged table.</li>
<li><span class="highlight">You created your union in a different database, and did not specify what database the tables are in</span><br />This particular situation caused me much grief. Consider that you are using an interactive MySQL client, and that you have selected the test2 database. Your source tables are in the test database. Now when you write your merge, you may do:
<pre class="prettyprint lang-sql">
ALTER TABLE `test`.`my_merged_table` ENGINE=MERGE UNION=(
`table1`,`table2`,`table3`
);
</pre>
<p>
You would expect MySQL to be smart enough to figure out that table1,table2,table3 are in the test database since that is where the merged table is being saved.
</p>
<p>Except, it doesn't. It assumes table1,table2,table3 refer to the current database, and will create the merged table like that.</p></li>
<li><span class="highlight">The table schemas are identical, but the binary versions are different.</span><br />e.g. created by a different version of MySQL. You can check this by querying the <tt>information_schema</tt> database: <pre class="prettyprint lang-sql">SELECT * FROM `information_schema`.`tables` WHERE `table_name`='one'\G
*************************** 1. row ***************************
TABLE_CATALOG: def
TABLE_SCHEMA: sync
TABLE_NAME: one
TABLE_TYPE: BASE TABLE
ENGINE: MyISAM
VERSION: 10
ROW_FORMAT: Fixed
TABLE_ROWS: 2000
AVG_ROW_LENGTH: 7
DATA_LENGTH: 14000
MAX_DATA_LENGTH: 1970324836974591
INDEX_LENGTH: 22528
DATA_FREE: 0
AUTO_INCREMENT: 2049
CREATE_TIME: 2018-09-13 19:02:01
UPDATE_TIME: 2018-09-13 19:02:01
CHECK_TIME: NULL
TABLE_COLLATION: utf8_general_ci
CHECKSUM: NULL
CREATE_OPTIONS:
TABLE_COMMENT:
1 row in set (0.00 sec)
</pre>
You can see the <span class="yellow-reverse">VERSION: 10</span> entry. All of the tables in your <tt>MERGE</tt> need to have identical versions.
</ol>
</p>
<p>Once you encounter this error on your MERGE table, MySQL clams shut and the only thing that you can now do is drop the table<a href="#ref1" id="1ref"><sup>1</sup></a>. It is not possible to inspect it via <tt>SHOW</tt> commands or even via the <tt>information_schema</tt>.</p>
<p>If you are not sure why you are getting this error, you may want to look at exactly how MySQL defined your merge. Since <pre class="prettyprint lang-sql">DESCRIBE</pre> and <pre class="prettyprint lang-sql">SHOW CREATE TABLE</pre> and the likes do not work on a busted merge table. Your only other option is to look at the physical merge definition file on the disk.</p>
<p>The file you are looking for is in the MySQL data directory, in the subdirectory of your database, and is called example.MRG.</p>
<p>The format of the file is plain ASCII and it just lists, one entry per line, the tables, in order, that are part of the union for that table</p>
<p>It is possible to edit this file<a href="#ref2" id="2ref"><sup>2</sup></a> and remove the entries MySQL complains about. After doing this, issue a <pre class="prettyprint lang-sql">FLUSH TABLE `thetable`;</pre> command, and you should now have a working MERGE table.</p>
<a id="ref1" href="#1ref"><sup>1</sup></a> Ok, I lied, you can still issue every conceived MySQL command against the table, it is just that MySQL will sound like a broken record and almost petulantly refuse to honour your wishes.<br>
<a id="ref2" href="#2ref"><sup>2</sup></a> Edit at your own risk, and preferably on the server itself via <tt>vi</tt>, <tt>emacs</tt>, or <tt>nano</tt>. I do not know how safe MySQL/MariaDB is against corruption in merge files, so <strong>always</strong> test offline first!Andy Pietershttp://www.blogger.com/profile/02919532566299594342noreply@blogger.com4